System and method for returning results of a query from one or more slave nodes to one or more master nodes of a database system

ABSTRACT

A database system for processing a query request comprises a master node that communicates a request to perform one or more activities associated with a precompiled query over at least one communication channel. A plurality of slave nodes are coupled to the master node. At least a particular one of the slave node receives the request communicated by the master node, performs at least a portion of the one or more activities associated with the request to obtain one or more results for the request, communicates a request-to-send message to the master node indicating that the one or more results are available for communication to the master node, and communicates at least a portion of the one or more results to the master node in response to receiving a permission-to-send message from the master node.

TECHNICAL FIELD OF THE INVENTION

This invention relates in general to computing systems, and moreparticularly to a system and method for returning results of a queryfrom one or more slave nodes to one or more master nodes of a databasesystem.

BACKGROUND

Conventional relational database systems are often capable of storing,organizing, and/or processing large amounts of data. As an example,relational database systems may be capable of storing, organizing,and/or processing many millions or billions of records. In thesesystems, data organization is vital to the processing efficiency of thedatabase system. Data organization within the relational database systemis particularly important in relational database systems that executerelatively complex queries and other commands involving relatively largeamounts of data.

In a typical relational database system, relationships are used tobreakdown the data into simpler structures for storage in one or moredata-storage devices. As a result, related information may be stored anddistributed over multiple data-storage devices. In most cases, before arelational database system can process a query, the relational databasesystem must redistribute the data so that it may be processed and/orcorrelated according to the received query.

SUMMARY OF EXAMPLE EMBODIMENTS

According to the present invention, certain disadvantages and problemsassociated with previous techniques for processing query requests in adatabase system may be reduced or eliminated.

In certain embodiments, a database system for processing a query requestcomprises a first master node operable to communicate a request toperform one or more activities associated with a precompiled query overat least one communication channel of the database system.

The system also includes a plurality of slave nodes coupled to the firstmaster node. At least a particular one of the slave nodes operable toreceive over the communication channel the request communicated by thefirst master node, perform at least a portion of the one or moreactivities associated with the request to obtain one or more results forthe request, communicate a request-to-send message to the first masternode indicating that the one or more results are available forcommunication to the first master node, and communicate at least aportion of the one or more results to the first master node in responseto receiving a permission-to-send message from the master node.

In certain other embodiments, a method for processing a query requestcomprises communicating from a first master node a request to performone or more activities associated with a precompiled query over at leastone communication channel of the database system. The method alsoincludes: receiving, at least at a particular one of a plurality ofslave nodes coupled to the first master node, over the communicationchannel the request communicated by the first master node; performing atthe particular slave node at least a portion of the one or moreactivities associated with the request to obtain one or more results forthe request; communicating from the particular slave node arequest-to-send message to the first master node indicating that the oneor more results are available for communication to the first masternode; and communicating from the particular slave node at least aportion of the one or more results to the first master node in responseto receiving a permission-to-send message from the master node.

Various embodiments may be capable of improving the processingefficiency of a database system. Certain embodiments may be capable ofprocessing query requests submitted to the database system using one ormore precompiled queries. In certain embodiments, one or more activitiesfor resolving at least a portion of a query request may performed by oneor more slave nodes having access to relevant data for resolving thatportion of the query request in a databases system, which may increasethe processing efficiency of the database system.

In certain embodiments, the present invention may reduce or eliminatethe possibility that more than one slave node will handle a request toperform an activity associated with a precompiled query communicated bythe master node. Certain embodiments of the present invention may becapable of managing throughput in the processing on the database systemof query requests received from one or more clients. In certainembodiments, prioritizing requests that are communicated to one or moreslave nodes by master nodes may help the database system to increase ormaximize throughput for query requests of higher importance or priority.In certain embodiments, the present invention may enable one or moremaster nodes of the database system to decrease or minimize the amountof time that the master nodes are idle waiting for one or more slavenodes to send results obtained for requests sent by the master nodes tothe slave nodes.

Certain embodiments of the present invention may provide some, all, ornone of the above technical advantages. Certain embodiments may provideone or more other technical advantages, one or more of which may bereadily apparent to those skilled in the art from the figures,descriptions, and claims included herein.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention and featuresand advantages thereof, reference is now made to the followingdescription, taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 illustrates an example system for processing query requestsaccording to one embodiment of the present invention;

FIG. 2 illustrates an example embodiment of a database system forprocessing one or more query requests associated with one or moreprecompiled queries deployed on one or more nodes of the databasesystem;

FIG. 3 illustrates an example query execution graph associated with anexample precompiled query;

FIG. 4 illustrates an example sorted table that includes a plurality ofkey parts;

FIG. 5 illustrates an example top level key associated with a sortedtable;

FIG. 6 illustrates an example method for processing one or more queryrequests in accordance with one embodiment of the present invention;

FIG. 7 illustrates an example method for processing a request to performan activity associated with a precompiled query communicated on acommunication channel by a master node and received on the communicationchannel by two or more slave nodes;

FIG. 8 illustrates an example method for managing the receipt andprocessing of query requests at one or more master nodes of the databasesystem;

FIG. 9 illustrates an example method for processing requests to performone or more activities associated with a precompiled query that arecommunicated by a particular master node according to prioritiesassigned to the requests; and

FIG. 10 illustrates an example method for returning results of a requestto perform one or more activities associated with a precompiled querycommunicated by a master node from one or more slave nodes to the masternode.

DESCRIPTION OF EXAMPLE EMBODIMENTS

One aspect of the present disclosure provides a system and method forprocessing query requests within a parallel-processing system. Inparticular, this disclosure provides an exemplary parallel-processingdatabase system capable of processing query requests. It should beappreciated that the certain concepts described within this disclosuremay apply or be implemented within any other parallel-processing systemwithout departing from the scope of the present disclosure. Moreover,particular examples specified throughout this document are intended forexemplary purposes only, and are not intended to limit the scope of thepresent disclosure.

FIG. 1 illustrates an example system 10 for processing query requestsaccording to one embodiment of the present invention. FIG. 1 illustratesjust one example embodiment of system 10. It should be appreciated thatother embodiments of system 10 may be used without departing from thescope of the present invention. In certain embodiments, system 10includes one or more clients 12 coupled to a first database system 14via a network 16. In general, clients 12 submit one or more queryrequests to database system 14, and database system 14 processes thequery requests using one or more precompiled queries distributed throughone or more nodes of database system 14.

As used throughout this document, the term “precompiled query” refers toa query that has been deployed on a database system (e.g., databasesystem 14) in advance of a user executing such query (i.e., bysubmitting a query request) on such database system. Additionally, asused throughout this document, the term “couple” and/or “coupled” refersto any direct or indirect communication between two or more elements,whether or not those elements are in physical contact with one another.

First database system 14 of system 10 is capable of performing one ormore desired computing and/or communicating functions. For example,first database system 14 may be capable of storing, organizing, sorting,processing, correlating, and/or communicating one or more keys and/orindices associated with one or more data files. In addition, firstdatabase system 14 may be capable of, for example, storing, deploying,processing, and/or executing one or more precompiled queries, andpre-keying data associated with the precompiled queries. Moreover, firstdatabase system 14 is operable to execute one or more precompiledqueries upon receiving a query request to execute a precompiled queryfrom a user of system 14 to resolve the received query request.

In certain embodiments, first database system 14 includes any device orcombination of devices that may include one or more hardware, software,and/or firmware modules. For example, first database system 14 mayinclude a parallel-processing databases' that includes a plurality offirst nodes 20 ₁-20 _(M) capable of storing, organizing, correlating,processing, and/or manipulating data. In various embodiments, each offirst nodes 20 may include or have access to, for example, one or moreprocessor modules, one or more memory modules, and/or one or moresoftware modules. In certain embodiments, as described in more detailbelow with reference to FIG. 2, each of nodes 20 may include, forexample, a master node, a slave node, or a combination of a master nodeand slave node. In this particular embodiment, each of nodes 20 operatesas both a master node and a slave node.

One or more of nodes 20 on first database system 14 may include or haveaccess to one or more precompiled queries 24. In certain embodiments,each of first nodes 20 ₁-20 _(M) includes or has access to one or moreprecompiled queries 24. Additionally, each precompiled query 24 may bedeployed to any or all of first nodes 20 ₁-20 _(M). In certainembodiments, a precompiled query 24 may be operable to resolve one ormore data requests received from a user of database system 14. Forexample, a precompiled query 24 may return one or more desired addressesor a range of addresses when any combination of one or more first names,one or more last names, or one or more social security numbers aresupplied in a query request to system 14. That is, when any of aplurality of variables is supplied in a request to the database system,precompiled query 24 enables system 14 to provide the appropriateaddress or addresses as an output.

As another example, precompiled query 24 is operable to return any oneof a plurality of desired responses or ranges of responses. In otherwords, when any of the plurality of variables is supplied in a queryrequest to database system 14, precompiled query 24 enables system 14 toprovide any one of a plurality of desired outputs or ranges of desiredoutputs. In one non-limiting example, precompiled query 24 may, whenquery request that includes any combination of variables is submitted tosystem 14, initiate the return of one or more desired responses orranges of responses. For example, precompiled query 24 may return allvehicle owners having a specific first name, such as Richard, when anycombination of one or more years a vehicle was produced, one or morevehicle makes, one or more vehicle models, one or more vehicle colors,one or more states of vehicle registration, one or more vehiclesuggested retail prices, and one or more zip codes that a vehicle wassold in of vehicle is supplied in a query request to system 14.

Each precompiled query 24 may include, for example, software, code,portions of code, data compilations, and/or a combination of these orany other type of data or executable. In certain embodiments, eachprecompiled query 24 includes an annotated query execution graph and oneor more dynamic link libraries (DLLs) capable of resolving future and/orroutine data requests on system 14, such as those received from a userof system 14.

Each precompiled query 24 may be associated with a series of actions oractivities for resolving a query request for invoking the precompiledquery 24. Activities may include, for example, one or more index reads,one or more de-duplications, one or more data sorts, one or more datarollups, or any other suitable activities according to particular needs.In certain embodiments, each precompiled query 24 is associated with aquery execution graph representing the series of actions or activitiesfor resolving a query request invoking the precompiled query 24. In somecases, precompiled query 24 was created using a programming languagecompiler, such as an ECL compiler, that converts a representation of adesired query into intermediary source code and/or a query executiongraph. In those cases, the programming language compiler may have mappedeach activity associated with the query execution graph to portions ofthe intermediary source code and/or one or more data files associatedwith the respective activity.

As an example, a representation of a query may have been converted intointermediary source code and a query execution graph, and theintermediary source code may have been compiled to generate one or moreexecutables in machine-level code. In some cases, the intermediarysource code may have been converted, using a programming languagecompiler, such as a C++ compiler, into the one or more executables. Invarious embodiments, the one or more executables may include, forexample, dynamically-linked executables, fully-linked executables, or ashared library. In this example, the one or more executables includeDLLs that are capable of being executed dynamically, in whole or inpart, by other executables.

In certain embodiments, each activity associated with the queryexecution graph may be annotated and mapped to one or more DLLs and/ordata files or tables associated with the respective activity. In somecases, one or more helper files capable of assisting in the processingof the annotated execution graph may have been created. The helper filesmay, for example, identify the appropriate DLLs and/or data files ortables for processing a particular activity associated with theannotated query execution graph. These helper files may be stored on orotherwise be accessible to nodes 20 of first database system 14.

Each activity associated with the query execution graph may be assigneda unique identification, which may be referred to as an activity ID.Each activity ID may uniquely identify the activity associated with theactivity ID with respect to other activities in the query executiongraph for a particular precompiled query 24. In certain embodiments, theactivity ID may include a first ID, which identifies the particularprecompiled query 24 with which the activity identified by the activityID is associated, and a second ID, which uniquely identifies theactivity identified by the activity ID with respect to other activitiesassociated with the particular precompiled query 24. As an example, ifthe query execution graph includes an index read, each activity of theindex read may be assigned a unique activity ID. In other cases, eachactivity may be mapped to a data file location within a database such asdatabase 14. In those cases, a table or index that identifies thelocation and/or content of data files or tables stored within a databasesystem such as database 14 may be used to perform the mapping.

Database system 14 may include one or more data tables that aredistributed over one or more of nodes 20, each portion of a data tablethat is stored on one or more of nodes 20 being stored as a key part 30.As used throughout this description, the term “table” refers to any datastructure, arrangement, or compilation of information. In some cases,each data table may be distributed across each of nodes 20 ₁-20 _(M).For example, a portion of each data table may be stored on each of nodes20 ₁-20 _(M) as key parts 30. In certain embodiments, each node 20 ofdatabase system 14 includes a portion of each key part 30. Each key part30 may be associated with one or more precompiled queries 24 and includedata for resolving a query request corresponding to the associatedprecompiled query 24.

The data stored within each key part 30 is typically sorted according toone or more precompiled queries 24. For example, when a precompiledquery 24 is created to return a desired response when any of a pluralityof variables is provided to first database system 14, the data in eachkey part 30 may be sorted according to various combinations of thevariables. As used through this description, the term “key part” refersto a portion of a sorted data file that includes data capable ofsatisfying and/or identifying the location of additional data thatsatisfies at least a portion of a precompiled query 24.

As one particular non-limiting example, a precompiled query 24 iscapable of returning a desired address when any of a combination offirst name, last name, or social security number is provided to firstdatabase system 14. In that case, the data stored in key parts 30 may besorted according to various combinations of the input variables. Forexample, the data in the tables may be sorted by first name, last name,and social security number; by social security number and address; or byany other appropriate combination. In some cases, a user may be able toidentify one or more combinations of input variables to be used inprocessing precompiled query 24.

Each key part 30 may be associated with a top level key 32 stored on oneor more of nodes 20. Top level keys operate to identify the locationwithin database system 14 of each key part 30. For example, wheredatabase system 14 includes a distributed table that is sorted by firstname, top level key 32 identifies the location of each key part 30 ofthe distributed table. Thus, if a first key part 30 was stored on node20 ₁ and included first names ranging from Aaron to Brian, and atwentieth key part 30 was stored on node 20 ₂₀ and included first namesranging from Frank to Gavin, then top level key 32 would identify thatfirst names ranging from Aaron to Brian are on node 20 ₁ and first namesranging from Frank to Gavin are on node 20 ₂₀. Top level keys 32 mayinclude any other suitable information, according to particular needs.

In some cases, top level key 32 and key parts 30 ₁-30 _(N) can includeadditional information to assist in identifying the appropriate locationof additional data. For example, if top level key 32 was created toidentify the location of key parts 30 associated with a distributedtable sorted by first name, then top level key 32 may include the firstname, last name, and address of the last record associated with therespective key part 30.

As such, first database system 14 may be pre-keyed (i.e., prior toprocessing query requests, such as requests to execute one or moreprecompiled queries 24). As used throughout this document, the term“pre-key” or “pre-keyed” refers to deploying one or more key parts, oneor more top level keys, and/or one or more indices on a database systemin advance of or simultaneously with a user requesting data from suchkey parts or indices (e.g., using a query request). One aspect of thisdisclosure recognizes that deploying one or more precompiled queries andpre-keying a database system can increase the processing efficiency ofthe database system for routine and/or standard data requests.

In certain embodiments, system 10 includes a second database system 40,which may be coupled to first database system via a link 42. In suchembodiments, second database system 40 may store the data that is to bethe subject of precompiled queries 24, prior to that data being storedon database system 14. Furthermore, in some cases, key parts 30 and toplevel keys 32 may be created using second database system 40 and latercommunicated to first database system 14. The following exampledescribes just one way of pre-keying first database system 14 usingsecond database system 40. The present invention contemplates pre-keyingfirst database system 14 in any suitable manner, with or without usingsecond database system 40.

In this example, second database system 40 is capable of performing oneor more desired computing and/or communicating functionalities. Forexample, second database system 40 may be capable of storing,organizing, sorting, distributing, processing, correlating, and/orcommunicating data associated with one or more raw data files. Seconddatabase system 40 may comprise any device or combination of devicesthat may include one or more hardware, software, and/or firmwaremodules. In this example, second database system 40 includes aparallel-processing database that includes a plurality of second nodes44 ₁-44 _(N) capable of storing, organizing, correlating, processing,distributing, and/or manipulating data. In various embodiments, each ofsecond nodes 44 may include or have access to, for example, one or moreprocessor modules, one or more memory modules, and/or one or moresoftware modules.

In certain embodiments, second database system 40 may include one ormore processors capable of managing the organization of the stored dataand coordinating the retrieval of the data from second nodes 44 ₁-44_(N) in response to queries, commands, and/or requests received bysecond database system 40. In various embodiments, second databasesystem 40 can receive requests, commands, and/or queries in a standardformat, such as structured query language (SQL), extensible markuplanguage (XML), hypertext markup language (HTML), or any other desiredformat.

In this example, a query representation may be communicated to seconddatabase system 40, from client 12 or from any other suitable source forexample and in any suitable manner. The query representation may be usedto generate a precompiled query 24 for the query representation. Thequery representation may include, for example, software, code, portionsof code, data compilations, and/or a combination of these or any othertype of data or executable. For example, the query representation mayinclude an HTML document that represents a query that is capable ofresolving future and/or routine data requests on system 14. In otherembodiments, the query representation could include, for example, an XMLdocument, a text file, or any other representation of the desired query.

In various embodiments, second database system 40 processes the queryrepresentation and performs certain computing functions to resolve oneor more activities associated with the query representation. In somecases, second database system 40 may process the query representationand perform one or more sorts on portions of the data stored on nodes 44₁-44 _(N). In those cases, the sorts of the data can result in theformation of one or more distributed tables of the data necessary toresolve at least a portion of the query representation. For example,where the query representation is created to return a desired responsewhen any of a plurality of variables is provided to first databasesystem 14, second database system 40 operates to sort the appropriatedata according to various combinations of the variables.

As one particular non-limiting example, the query representation iscapable of returning a desired address when any combination of firstname, last name, or social security number is provided to first databasesystem 14. In that case, second database system 40 processes the queryrepresentation and sorts the data according to various combinations ofthe input variables. For example, second database system 40 may sort thedata by first name, last name, and address; by last name and address; bysocial security number and address; or by any other appropriatecombination.

In most cases, the one or more sorts of the data within second databasesystem 40 create one or more tables that are distributed over one ormore of second nodes 44 ₁-44 _(N). In some cases, the sort of the datawithin second database system 40 creates a table that is distributedover each of nodes 44 ₁-44 _(N). In various embodiments, each of secondnodes 44 ₁-44 _(N) that receives a portion of the distributed tablestores that portion as a key part 30.

Moreover, the sort of the data within second database system 40 alsogenerates a top level key 32 for each table. Top level key 32 operatesto identify the location within database system 40 of each key part 30associated with the respective table. For example, where second databasesystem 40 generates a distributed table that is sorted by first name,top level key 32 identifies the location of each key part 30 ₁-30 _(N)of the distributed table. Thus, if a first key part 30 ₁ was stored onnode 44 ₁ and included first names ranging from Aaron to Brian, and atwentieth key part 30 ₂₀ was stored on node 44 ₂₀ and included firstnames ranging from Frank to Gavin, then top level key 32 would identifythat first names ranging from Aaron to Brian are on node 44 ₁ and firstnames ranging from Frank to Gavin are on node 44 ₂₀.

In some cases, top level key 32 and key parts 30 ₁-30 _(N) can includeadditional information to assist in identifying the appropriate locationof additional data. For example, if top level key 32 is created toidentify the location of key parts 30 associated with a distributedtable sorted by first name, then the top level key 32 may include thefirst name, last name, and address of the last record associated withthe respective key part 30.

In this particular embodiment, after creation of key parts 30 ₁-30 _(N)and top level key 32, second database system operates to pre-key firstdatabase system 14. For example, a precompiled query 24 may becommunicated to first database system 14, originating from client 12 asa query representation for example. In certain embodiments, firstdatabase system 14 deploys precompiled query 24 on at least one of firstnodes 20. In other embodiments, first database system 14 deploysprecompiled query 24 on each of first nodes 20 ₁-20 _(M). In thisparticular example, precompiled query 24 is deployed on node 20 ₁. Inthat example, node 20 ₁ distributes a copy of precompiled query 24 toeach of nodes 20 ₂-20 _(M). Although precompiled query 24 is deployed tonode 20 ₁ in this example, precompiled query 24 may be deployed to anyor all of nodes 20 ₁-20 _(M) without departing from the scope of thepresent disclosure.

In this particular embodiment, node 20 ₁ operates to read the annotatedquery execution graph associated with precompiled query 24 and identifyone or more data files or tables necessary to satisfy a particularactivity of precompiled query 24. Although node 20 ₁ operates to readthe query execution graph and identify one or more data files or tablesin this example, any or all of node 20 ₁-20 _(M) can perform the desiredfunctionality without departing from the scope of the present invention.In some cases, node 20 ₁ can identify the one or more data files ortables using the mapping of each activity to the one or more DLL'sand/or data files or tables. In other cases, node 20 ₁ can identify theone or more data files or tables using the one or more helper filesassociated with precompiled query 24. In various embodiments, node 20 ₁may be capable of generating and/or communicating one or more datarequests to acquire the necessary data from its permanent location, suchas a location within a database system.

In this example, node 20 ₁ of first database system 14 communicates oneor more requests to acquire the necessary data from second nodes 44 ₁-44_(N) of second database system 40. Node 20 ₁ communicates the one ormore requests to second database system 40 through communications link42. In this example, second database system 40 receives and processesthe one or more requests to communicate data necessary to resolveprecompiled query 24. Second database system 40 then operates to pre-keyfirst database system 14 by communicating copies of key parts 30 ₁-30_(N) and top level key 32 associated with each sorted table to firstdatabase system 14.

Unlike conventional database systems that typically combine all the keyparts into a single key or index, first database system 14 stores eachindividual key part 30 separately on the appropriate first node 20. Inthis example, first database system 14 distributes key parts 30 storedon second nodes 44 over first nodes 20 ₁-20 _(M). In some cases, thenumber of first nodes 20 of first database system 14 can be differentthan the number of second nodes 44 of second database system 40. In thisparticular embodiment, the number of first nodes 20 is less than thenumber of second nodes 44. Thus, at least some of nodes 20 ₁-20 _(M) maystore more than one key part 30. In various embodiments, each of keyparts 30 ₁-30 _(N) may be stored on more than one of first nodes 20 ₁-20_(M). One aspect of this disclosure recognizes that, in certainembodiments, storing copies of each key part 30 on multiple first nodes20 enhances the systems reliability by providing redundancy which canminimize the effects of a single failure of a first node 20 on firstdatabase system 14.

In one non-limiting example, second database system 40 includesfour-hundred second nodes 44 ₁-44 ₄₀₀ and first database system 14includes forty first nodes 20 ₁-20 ₄₀. Although this example implementsfour-hundred second nodes 44 and forty first nodes 20, any number ofsecond nodes 44 and first nodes 20 can be used without departing fromthe scope of the present disclosure. In that example, if each of secondnodes 44 ₁-44 ₄₀₀ store a respective key part 30 ₁-30 ₄₀₀ associatedwith a sorted table, then first database system 14 would distribute eachof those four-hundred key parts 30 ₁-30 ₄₀₀ over first nodes 20 ₁-20 ₄₀.In various embodiments, first database system 14 could distribute keyparts 30 ₁-30 ₄₀₀ such that first node 20 ₁ receives key parts 30 ₁, 30₄₁, 30 ₈₁, . . . 30 ₂₄₁, from second nodes 44 ₁, 44 ₄₁, 44 ₈₁, . . . 44₂₄₁, and first node 20 ₄₀ receives key parts 30 ₄₀, 30 ₈₀, 30 ₁₂₀, . . .30 ₄₀₀ from second nodes 44 ₄₀, 44 ₈₀, 44 ₁₂₀, . . . 44 ₄₀₀. In someembodiments, first database system 14 could distribute key parts 30 ₁-30₄₀₀ such that first node 20 ₁ receives key parts 30 ₁-30 ₁₀ from secondnodes 44 ₁-44 ₁₀, and first node 20 ₄₀ receives key parts 30 ₃₉₁-30 ₄₀₀from second nodes 44 ₃₉₁-44 ₄₀₀. In other embodiments, first databasesystem 14 could distribute key parts 30 ₁-30 ₄₀₀ in any other suitablemanner.

In this example, second database system 40 also communicates top levelkeys 32 to node 20 ₁ of first database system 14. In other embodiments,second database system 40 can communicate top level keys 32 to any orall of nodes 20 ₁-20 _(M). In this particular embodiment, first node 20₁ distributes top level key 32 to each of first nodes 20 ₁-20 _(M). Inother embodiments, any one of first nodes 20 may be capable ofdistributing top level key 32 to each of first nodes 20 ₁-20 _(M).

In this particular embodiment, system 14 operates to map the location ofeach key part 30 ₁-30 _(N) from its respective second node 44 to one ormore communication channels associated with first database system 14. Inthis example, a particular node 20 that is processing a request toexecute a particular precompiled query 24 operates to map the locationof each key part 30 ₁-30 _(N) from its respective second node 44 to oneor more communication channels associated with first database system 14.In some cases, the particular node 20 may have access to one or morehelper files that may assist in mapping the location of each key part 30₁-30 _(N) to one or more communication channels. In this example, eachfirst node 20 has access to a function capable of mapping the locationof key parts 134 to one or more channel numbers associated with databasesystem 14. For example, the function may comprise “part_no MODnum_channels,” where MOD represents the modulus operation, part_norepresents the part number retrieved for the top level key 32, andnum_channels represents the number of communication channels.

In one non-limiting example, second database system 40 includesfour-hundred second nodes 44 ₁-44 ₄₀₀ and first database system 14includes forty first nodes 20 ₁-20 ₄₀. First database system 14 alsoincludes forty communication channels capable of carrying a multicastcommunication signal to one or more first nodes 20 ₁-20 ₄₀. Althoughthis example implements forty communication channels within firstdatabase system 14, any number of communication channels can be usedwithout departing from the scope of the present disclosure. In thatexample, if each of second nodes 40 ₁-40 ₄₀₀ store a respective key part30 ₁-30 ₄₀₀ associated with a sorted table, then first database system14 may distribute each of those four-hundred key parts 30 ₁-30 ₄₀₀ overfirst nodes 20 ₁-20 ₄₀. In various embodiments, first database system 14could distribute key parts 30 ₁-30 ₄₀₀ such that first node 20 ₁receives key parts 30 ₁, 30 ₄₁, 30 ₈₁, . . . 30 ₃₆₁, from second nodes40 ₁, 40 ₄₁, 40 ₈₁, . . . 40 ₃₆₁, and first node 20 ₄₀ receives keyparts 30 ₄₀, 30 ₈₀, 30 ₁₂₀, . . . 30 ₄₀₀ from second nodes 40 ₄₀, 40 ₈₀,40 ₁₂₀, . . . 40 ₄₀₀. Moreover, first database system 14 coulddistribute key parts 30 ₁-30 ₄₀₀ such that first node 20 ₁ also receiveskey parts 30 ₄₀, 30 ₈₀, 30 ₁₂₀, . . . 30 ₄₀₀ from second nodes 40 ₄₀, 40₈₀, 40 ₁₂₀, . . . 40 ₄₀₀ to add redundancy to system 14. In someembodiments, first database system 14 could distribute key parts 30 ₁-30₄₀₀ such that first node 20 ₁ receives key parts 30 ₁-30 ₁₀ from secondnodes 40 ₁-40 ₁₀, and first node 20 ₄₀ receives key parts 30 ₃₉₁-30 ₄₀₀from second nodes 40 ₃₉₁-40 ₄₀₀. In other embodiments, first databasesystem 14 could distribute key parts 30 ₁-30 ₄₀₀ in any other suitablemanner.

In this particular non-limiting example, first node 20 ₁ receives aquery request from a user (e.g., a user of client system 12) to executea particular precompiled query 24 ₉. In processing precompiled query 24₉, node 20 ₁ identifies an activity that necessitates the retrieval of akey part 30 ₇₇ stored at least on nodes 20 ₁₇ and 20 ₂₇. First node 20 ₁accesses the appropriate top level key 32 for precompiled query 24 ₉ andmaps the location of key part 30 ₇₇ from second node 44 ₇₇ to aparticular communication channel, such as channel fifteen. In thisparticular embodiment, both of nodes 20 ₁₇ and 20 ₂₇ are capable ofreceiving one or more requests at least on communication channelfifteen.

Although formation of key parts 30 and top level keys 32 has beendescribed in a particular manner (i.e., using second database system40), the present invention contemplates forming key parts 30 and toplevel keys 32 in any suitable manner, according to particular needs. Asjust one example, key parts 30 and top level keys 32 may be dynamicallycreated as one or more query requests are received by database system14, from client 12 for example. Furthermore, although second databasesystem 40 has been described as storing the data that is to be thesubject of precompiled queries 24, prior to that data being stored ondatabase system 14, the present invention contemplates database system14 acquiring the data that is to be the subject of precompiled queries24 from any suitable source, according to particular needs. Moreover, inthe described embodiment of system 10, each of first database system 14and second database system 40 includes a separate database system. In analternative embodiment, first database system 14 and second databasesystem 40 could each be part of a common larger database system.Moreover, each of first nodes 20 could form part of second nodes 44.

In certain embodiments, one or more clients 12 couple to system 10through network 16. Each client 12 may include any computing and/orcommunication device operable to communicate and/or receive information.For example, each client 12 may include a web server, a work station, amainframe computer, a mini-frame computer, a desktop computer, a laptopcomputer, a personal digital assistant, a wireless device, and/or anyother computing or communicating device or combination of devices. Inoperation, each client 12 may execute with any of the well-known MS-DOS,PC-DOS, OS-2, MAC-OS, WINDOWS™, UNIX, or other appropriate operatingsystems. Moreover, “client 12” and “user of client 12” may be usedinterchangeably without departing from the scope of this invention.Although a single client 12 is illustrated, the present inventioncontemplates any suitable number of clients 12 being coupled to databasesystem 14, and database system 40 where appropriate.

In certain embodiments, client 12 includes a query submission module 50and a graphical user interface (GUI) 52 that enable a user to submit oneor more query requests 54 for processing by database system 14 and toview results of the submitted query requests 54 returned by databasesystem 14. For example, client 12 may submit one or more query requests54 using query submission module 50. In some cases, query submissionmodule 50 and GUI 52 enable a user to submit a query request 54 thatincludes one or more data requests on system 14. For example, a user ofclient 12 may submit a query request 54 that specifies one or morevariables for retrieval of a result based on the one or more variables.In a particular example, a user of client 12 may submit a query request54 for returning a desired address when any combination of a first name,a last name, or a social security number is supplied in a query request54 to system 14. That is, when any of the plurality of variables issupplied in a query request 54 to database system 14, query request 54prompts system 14 to provide the appropriate address as an output.

In certain embodiments, query request 54 corresponds to one or more ofprecompiled queries 24 ₁-24 _(w) stored on one or more nodes 20 ofsystem 14. For example, query request 54 may include a request forsystem 14 to initiate an instance of a precompiled query 24 based on theone or more variables submitted in the query request 54. Although queryrequests 54 that correspond to a precompiled query 24 stored on system14 are primarily described, the present invention contemplates receivingand processing query requests 54 that do not have a correspondingprecompiled query 24. Such query requests lacking a correspondingprecompiled query 24 may be processed by, for example, dynamicallycreating in any suitable manner one or more key parts 30 andcorresponding top level key 32 for resolving the query request 54.

Query submission module 50 may include any device or combination ofdevices that may include one or more hardware, software, and/or firmwaremodules. In certain embodiments, query submission module 50 includes,for example, software capable of being executed on client 12. In certainembodiments, query submission module 50 may include the necessaryhardware, software, and/or firmware capable of providing an XML or HTMLtemplate for display on GUI 52, on a web browser for example.

In certain embodiments, query submission, module 50 may display a formthat includes one or more fields in which a user of client 12 may insertor select one or more search terms (i.e., input variables) as part ofquery request 54. Query request 54 may identify, be linked to, orotherwise be associated with one or more precompiled queries 24 storedon database system 14. As just one example, a user may provide or selectinput variables that include any combination of a first name, last name,or a social security number as part of query request 54, requestingdatabase system 14 to return one or more desired addresses for theprovided input variables.

In certain embodiments, query submission module 50 and/or GUI 52 enablesa user of client 12 to submit a desired query request 54 that isprecompiled in one or more query programming languages. The programminglanguage can include, for example, C++, Enterprise Control Language(ECL), Simple Query Language (SQL), Perl, or a combination of these orother programming languages. A query request 54 submitted using client12 may be in any suitable format, including XML, hypertext transferprotocol (HTTP), ECL, or any other suitable format.

Network 16 may include any wireless network, wireline network, orcombination of wireless and wireline networks capable of supportingcommunication between network elements using ground-based and/orspace-based components. For example, network 16 may include a datanetwork, a public switched telephone network (PSTN), an integratedservices digital network (ISDN), a local area network (LAN), a wide areanetwork (WAN), a metropolitan area network (MAN), all or a portion ofthe global computer network known as the Internet, and/or othercommunication systems or combination of communication systems at one ormore locations.

In certain embodiments, system 10 includes a query receiving module 60for receiving query requests 54 submitted to system 14 by client 12. Incertain embodiments, network 16 couples to client 12 through acommunications link 62 and to query receiving module 60 through acommunications link 64. In certain embodiments, query receiving module60 operates to route or otherwise direct query requests 54 submitted byclient to appropriate components within system 14. First database system14 may be coupled to query receiving module 60 via a link 66, and queryreceiving module 60 may be separate from database system 14.Alternatively, query receiving module 60 may a part of system 14. Queryreceiving module 60 may include any device or combination of devicesthat may include one or more hardware, software, and/or firmwaremodules.

Query receiving module 60 may route query requests received from client12 to one or more suitable components within database system 14 in anysuitable manner. For example, in certain embodiments, query receivingmodule 60 may route query requests to one or more of nodes 20 ₁-20 _(M).Query receiving module 60 may include a router or other suitablecomponent for routing query requests 54 to one or more suitablecomponents within database system 14, such as nodes 20. Query receivingmodule 60 may include load balancing capabilities, as described in moredetail below.

In the illustrated embodiment, system 10 includes at least communicationlinks 42, 62, 64, and 66 each operable to facilitate the communicationof data and/or queries within system 10. Communications links 42, 62,64, and 66 may include any hardware, software, firmware, or combinationthereof. In various embodiments, communications links 42, 62, 64, and 66may comprise communications media capable of assisting in thecommunication of analog and/or digital signals. Communications links 42,62, 64, and 66 may, for example, comprise a twisted-pair coppertelephone line, a fiber optic line, a Digital Subscriber Line (DSL), awireless link, a USB bus, a PCI bus, an Ethernet interface, or any othersuitable interface operable to assist in the communication within system10.

FIG. 2 illustrates an example embodiment of database system 14 forprocessing one or more query requests 54 associated with one or moreprecompiled queries 118 ₁-118 _(W) deployed on one or more nodes 20 ofdatabase system 14. In this example, database system 14 storesprecompiled queries 24 ₁-24 _(W), key parts 30 ₁-30 _(N) associated withone or more precompiled queries 24 ₁-24 _(W), and one or more top levelkeys 32 ₁-32 _(x) associated with precompiled queries 24 ₁-24 _(W).Moreover, first database system 14 is operable to execute one or moreprecompiled queries 24 ₁-24 _(W) upon receiving a query request 54 froma user of system 14, requesting system 14 to execute a particular one ormore precompiled queries 24. In this particular embodiment, each ofprecompiled queries 24 ₁-24 _(W) includes an annotated query executiongraph, one or more DLL's, and/or one or more helper files.

In this particular embodiment, each of precompiled queries 24 ₁-24 _(W)is capable of resolving one or more routine and/or standard datarequests that may have variations in input variables. For example,precompiled query 24 ₂ may be capable of returning one or more desiredaddresses or range of addresses when any combination of one or morefirst names, one or more last names, or one or more social securitynumbers are provided to first database system 14 (e.g., in a queryrequest 54), while precompiled query 24 ₃ may be capable of returning afirst name, last name, and state of registration, for all owners of oneor more specific make, model, year, and/or color of one or more vehicles(e.g., when so requested in a query request 54). One aspect of thisdisclosure recognizes that, in certain embodiments, deployingprecompiled queries 24 ₁-24 _(W) may increase the processing efficiencyof first database system 14 for routine and/or standard data requeststhat have a number of variations on the input parameters.

In this example, database system 14 comprises a parallel-processingdatabase that includes a plurality of nodes 20 ₁-20 _(M). In certainembodiments, each of nodes 20 of first database system 14 includes amaster node 70, a slave node 72, and one or more memory modules 74.Although each of nodes 20 ₁-20 _(M) includes master node 70, slave node72, and memory module 74 in this example, each of nodes 20 may includeany other appropriate device, or may exclude one or more of master node70, slave node 72, or memory module 74 without departing from the scopeof the present disclosure. Although an embodiment of system in whicheach of nodes 20 operates as both a master node and a slave node isprimarily described, it should be understood that each node 20 mayoperate as a master node, a slave node, or a combination of a masternode and slave node. In this particular embodiment, the number of masternodes 70 is the same as the number of slave nodes 72. In someembodiments, the number of slave nodes 72 can be larger than the numberof master nodes 70. In other embodiments, the number of master nodes 70can be larger than the number of slave nodes 72.

In this particular embodiment, precompiled queries 24 ₁-24 _(W) and toplevel keys 32 ₁-32 _(X) have been deployed to and stored on each ofmaster nodes 70 ₁-70 _(M). In some cases, a particular precompiled query24 and top level keys 32 associated with the particular precompiledquery 24 may be deployed to and stored on one of master nodes 70 ₁-70_(M). In that example, the master node 70 that receives the particularprecompiled query 24 and associated top level keys 32 operates todistribute a copy of the respective precompiled query 24 and top levelkeys 32 to each of the other master nodes 70 ₁-70 _(M). In otherembodiments, each of precompiled queries 24 ₁-24 _(W) and top level keys32 ₁-32 _(X) are distributed to and stored on each of master nodes 70₁-70 _(M). In certain embodiments, a copy of precompiled queries 24 ₁-24_(W) is stored on each of slave nodes 72 ₁-72 _(M). For example, one ormore of master nodes 70 may have distributed precompiled queries 24 ₁-24_(N) to slave nodes 72 ₁-72 _(M).

In this example, each master node 70 ₁-70 _(M) is capable of executingeach of precompiled queries 24 ₁-24 _(W) upon receiving a query request54 from a user of system 14, such as client 12. Moreover, each masternode 70 ₁-70 _(M) is capable of communicating a request to perform aparticular activity associated with a particular precompiled query 24,such as, a request to perform an index read, to one or more slave nodes72 ₁-72 _(M) (e.g., on the same or on different node 20 as the masternode 70 ₁-70 _(M)) for processing in accordance with one or more ofprecompiled queries 24 ₁-24 _(W). In some cases, the request to performa particular activity can include, for example, one or more variablesassociated with a request supplied by a user of system 14. In variousembodiments, each master node 70 ₁-70 _(M) is capable of communicatingthe request to perform a particular activity using a multicast signalformatted in, for example, User Datagram Protocol (UDP).

Master nodes 70 ₁-70 _(M) may comprise any device or combination ofdevices that may include one or more hardware, software, and/or firmwaremodules. In this particular embodiment, each master node 70 ₁-70 _(M)includes or has access to a memory that stores each precompiled query 24₁-24 _(W) deployed on system 14 and each top level key 32 ₁-32 _(X)associated with each deployed precompiled query 24 ₁-24 _(W).

In this example, each slave node 72 ₁-72 _(M) is capable of storing eachprecompiled query 24 received from a particular master node 70. Inaddition, each of slave nodes 72 ₁-72 _(M) is capable of processing oneor more requests to perform one or more particular activities associatedwith a specific precompiled query 24. Slave nodes 72 ₁-72 _(M) maycomprise any device or combination of devices that may include one ormore hardware, software, and/or firmware modules. In some cases, each ofslave nodes 72 ₁-72 _(M) may have access to and/or include one or morehelper files that may assist each of slave nodes 72 ₁-72 _(M) inprocessing a request to perform one or more particular activities.

In this example, each of slave nodes 72 ₁-72 _(M) has access to one ormore memory modules 74 capable of storing one or more key parts 30. Inother embodiments, each of slave nodes 72 ₁-72 _(M) may include one ormore memory modules 74. Memory modules 74 may include any hardware,software, firmware, or combination thereof operable to store andfacilitate retrieval of information. Each memory module 74 may storeinformation using any of a variety of data structures, arrangements,and/or compilations. Memory module 74 may, for example, include a harddisk, a dynamic random access memory (DRAM), a static random accessmemory (SRAM), or any other suitable volatile or nonvolatile storage andretrieval device or combination of devices.

In this particular embodiment, each of slave nodes 72 ₁-72 _(M) storesand provides access to key parts 30 associated with at least another oneof slave nodes 72 ₁-72 _(M). Moreover, each of slave nodes 72 ₁-72 _(M)is capable of receiving multicast signals on more than one communicationchannel. For example, slave node 72 ₂ operates to receive multicastcommunication signals on communication channels one and two, and storesand provides access to key parts 30 ₁ and 30 ₂. Meanwhile, slave node 72₃ operates receive multicast communication signals from communicationchannels two and three, and stores and provides access to key parts 30 ₂and 30 ₃. Thus, slave nodes 72 ₂ and 72 ₃ each receive a multicastsignal communicated on communication channel two and provide access todata associated with key parts 30 ₂ and thereby add redundancy to system14. In certain embodiments, each of slave nodes 72 ₁-72 _(M) registerswith one or more of the communication channels in order to receivemulticast signals sent on the one or more communication channels. Forexample, slave nodes 72 may register with certain communication channelsbased on the key parts 30 to which the slave nodes have access. Asanother example, slave nodes 72 may be assigned to register with certaincommunication channels in any suitable manner.

In various embodiments, system 14 is configured such that each of slavenodes 72 ₁-72 _(M) that receive multicast signals from a particularcommunication channel and that store and/or provide access to aparticular key part 30 are not susceptible to a single point of failure.One aspect of this disclosure recognizes that, in certain embodiments,storing copies of each key part 30 on multiple slave nodes 72 that arecapable of receiving a multicast signal on a particular communicationchannel enhances the systems reliability by providing system redundancy.In most cases, the provision of system redundancy can minimize theeffects of a single failure of a particular slave node 72 on firstdatabase system 14. Moreover, processing of requests communicated bymaster nodes 70 may be divided between multiple slave nodes 72. Forexample, if one or more master nodes 70 are communicating requests toperform index reads of a particular key part 30, more than one slavenode 72 may receive and process the index read because more than oneslave node 72 has access to the particular key part 30. This may allowthe processing load for requests involving a particular key part 30 tobe spread between or among multiple slave nodes 72.

In this example, a network 80 couples each of nodes 20 to each other.Network 80 may include any wireless network, wireline network, orcombination of wireless and wireline networks capable of supportingcommunication between network elements. For example, network 80 maycomprise a data network, a public switched telephone network (PSTN), anintegrated services digital network (ISDN), a local area network (LAN),a wide area network (WAN), a metropolitan area network (MAN), all or aportion of the global computer network known as the Internet, and/orother communication systems or combination of communication systems atone or more locations. In various embodiments, network 80 is capable oftransmitting information from master nodes 70 ₁-70 _(M) to one or moreslave nodes 72 ₁-72 _(M) over a plurality of communication channels.

In one non-limiting example, system 14 includes forty master nodes 70₁-70 ₄₀ and forty slave nodes 72 ₁-72 ₄₀. Although this example includesforty master nodes 70 and slave nodes 72, any number of master nodes 70and slave nodes 72 may be used without departing from the scope of thepresent disclosure. Moreover, in this example, system 14 includes or hasaccess to forty communication channels. In other embodiments, system 14may include, for example, two communication channels, ten communicationchannels, twenty communication channels, or more.

As described above with reference to FIG. 1, one or more clients 12 maycouple to system 14 through network 16. Although a single client 12 isillustrated, the present invention contemplates any suitable number ofclients 12 being coupled to database system 14. Each client 12 mayinclude substantially similar components and be capable of substantiallysimilar functionality to that described above with reference to FIG. 1.Clients 12 may submit one or more query requests 54 to database system14 for processing by system 14, using query submission module 50 and GUI52 for example. Clients 12 may also view results of the submitted queryrequests 54 returned by database system 14, using one or more of querysubmission module 50 and GUI 52 for example.

In some cases, query submission module 50 and GUI 52 enable a user tosubmit a query request 54 that includes one or more data requests onsystem 14. For example, a user of client 12 may submit a query request54 that specifies one or more variables for retrieval of a result basedon the one or more variables. In a particular example, a user of client12 may submit a query request for returning a desired address when anycombination of a first name, a last name, or a social security number issupplied in a query request 54 to system 14. That is, when any of theplurality of variables is supplied in a query request 54 to databasesystem 14, query request 54 enables system 14 to provide the appropriateaddress as an output.

In certain embodiments, query request 54 corresponds to one or more ofprecompiled queries 24 ₁-24 _(w) stored on one or more nodes 20 ofsystem 14. For example, the query request 54 include a request forsystem 14 to initiate an instance of a precompiled query 24 based on theone or more variables submitted in the query request 54. Although queryrequests 54 that correspond to a precompiled query 24 stored on system14 are primarily described, the present invention contemplates receivingand processing query requests 54 that do not have a correspondingprecompiled query 24. Such query requests lacking a correspondingprecompiled query 24 may be processed by, for example, dynamicallybuilding in any suitable manner one or more key parts 30 for resolvingthe query request 54.

As described above with reference to FIG. 1, in certain embodiments,system 10 includes a query receiving module 60 for receiving queryrequests 54 submitted by client 12 to system 14. Query receiving module60 may operate to route or otherwise direct query requests 54 submittedby client 12 to appropriate components within system 10. Query receivingmodule 60 may route query requests 54 received from client 12 to one ormore suitable components within database system 14 in any suitablemanner. For example, in certain embodiments, query receiving module 60may route query requests 54 to one or more of master nodes 70 ₁-70 _(M).Query receiving module 60 may include a router or other suitablecomponent for routing query requests 54 to one or more suitablecomponents within database system 14, such as master nodes 70.

In certain embodiments, master nodes 70 receive and assume primaryresponsibility for processing query requests 54 received by system 14.One or more master nodes 70 may receive query request 54, although itmay be desirable for only one master node 70 to receive query request54. In certain embodiments, query requests 54 may be communicated tomaster nodes 70 in a manner for optimizing load balancing between masternodes 70. For example, query receiving module 60 may include loadbalancing capabilities and may route query requests 54 to particularmaster nodes 70 in a manner that optimizes the processing load of masternodes 70.

In alternative embodiments, particular master nodes may be pre-assignedto one or more clients 12 for handling query requests received fromthose clients 12. In such embodiments, query receiving module 60 mayroute query requests 54 to particular master nodes 70 based on theclients 12 that submitted the query requests 54. Although queryreceiving module 60 is described, the present invention contemplatessystem 14 receiving query requests 54 from client 12 and routing thosequery requests 54 to appropriate master nodes 70 in any suitable manneraccording to particular needs, with or without query receiving module60. Furthermore, it may also be desirable to limit or otherwise controlthe number of queries that a particular client 12 may have running onsystem 14 at a particular time.

In certain embodiments, each master node 70 may be operable to receiveand process a predetermined number of query requests 54 substantiallyconcurrently. For example, a master node 70 may include a particularnumber of threads, each operable to receive and process a differentquery request 54. It should be understood that “different” in thiscontext does not necessarily mean that the information sought by thequery request 54 or the parameters provided in the query request 54(e.g., by a client 12) are different. For example, each thread may beoperable to receive and process a different instance of the same queryrequest 54. It may be desirable to assign, either exclusively ornon-exclusively, a particular number of these threads to each particularclient 12. As an example, a particular master node 70 may include thirtythreads, twenty of which are available to a first client 12 and ten ofwhich are available to a second client 12. In certain embodiments ofthis example, when the first client 12 is using all of twenty of itsthreads on the particular master node 70, the client 12 may be directedto another master node 70 or denied access to system 14 until such timeas one of the twenty nodes available to the first client 12 becomeavailable. Additionally or alternatively, if the second client 12 is notcurrently using the ten threads of the particular master node 70assigned to the second client 12 such that one or more of the tenthreads are currently idle, then the an operating system (OS) schedulermay adjust the central processing unit (CPU) cycles such that no orfewer clock cycles are used on the idle threads. As a particularexample, if the first client 12 is currently using all twenty of itsavailable threads and the second client 12 is not currently using any ofits ten threads, all CPU clock cycles may be used by the twenty threadsassigned to the first client 12, thereby increasing the speed at whichquery requests 54 submitted by the first client 12 are processed. Theseand other techniques may help to manage and/or increase throughput inprocessing of query requests 54 submitted to system 14.

Additionally, in certain embodiments, if no threads are available on aparticular master node 70 for a particular client 12 to submit queryrequests 54, the particular client 12 may be notified that theparticular master node 70 is too busy. The particular client 12 may thenselect a different master node 70 for submitting query requests 54 ormay pause and retry submitting query requests 54 on the particularmaster node 70. For example, system 14 may prompt the particular client12 to either select a different master node 70 or retry the particularmaster node 70 after a suitable delay.

Although the above example describes first and second clients 12, anysuitable number of clients 12 may be coupled to system 14, and one ormore threads of one or more master nodes 70 may be assigned to eachclient in any suitable manner. If the second client 12 later beginssending query requests 54 to system 14 for processing by the threadsassigned to the second client 12, at least the threads being used by thesecond client 12 may then be given adequate CPU cycles for processingthe query requests 54 submitted by the second client 12. Furthermore, incertain embodiments, query receiving module 60 may be operable tocommunicate query requests 54 to appropriate one or more appropriatemaster nodes 70 according to the above-described techniques. Moreover,the techniques for managing the threads of a master node 70 may helpsystem 14 to manage throughput in processing of query requests 54received from one or more clients 12. As used throughout thisdescription, managing throughput may include increasing throughput,substantially maximizing throughput, holding throughput substantiallyconstant, decreasing throughput, or substantially minimizing throughput;however, it may be desirable to increase throughput to the fullestextent possible.

Upon receiving a query request 54 (e.g., from query receiving module 60or otherwise), a master node 70 may process query request 54 todetermine if query request 54 corresponds to one of precompiled queries24 ₁-24 _(W). If a query request 54 does not correspond to one ofprecompiled queries 24 ₁-24 _(W), master node 70 may, in certainembodiments, initiate dynamic creation of one or more keys parts 42 forresolving query 22. If query request 54 does correspond to one or moreof precompiled queries 24 ₁-24 _(W), master node 70 is operable toinitiate processing of the one or more corresponding precompiled queries32.

A receiving master node 70 may review the annotated query executiongraph corresponding to the corresponding precompiled query 24 todetermine if one or more activities in the query execution graph requirea remote activity (i.e., an activity for processing at one or more ofslave nodes 72 ₁-72 _(M)). In some cases, master node 70 determineswhether the annotated query execution graph calls for one or more remoteactivities by reviewing the activity IDs assigned to each activity forthe corresponding precompiled query 24. If the corresponding precompiledquery 24 does not require any remote activities, master node 70 mayprocess each activity of the query execution graph for the correspondingprecompiled query 24. If the corresponding precompiled query 24 requiresone or more remote activities, master node 70 is operable to communicatea portion of the corresponding precompiled query 24, such as the remoteactivity, along with any other suitable information, such as one or moreinput variables specified in query request 54, to one or more slavenodes 72 ₁-72 _(M) for processing. The remote activity may include, forexample, an index read, a record read, an aggregation, or any otheractivity that necessitates the use of one or more slave nodes 72 ₁-72_(M).

Master node 70 is operable to determine the appropriate one or moreslave nodes 72 for handling the remote activity of the correspondingprecompiled query 24. In certain embodiments, master node 70 may notactually determine the one or more slave nodes 72 for handling theremote activity, but may instead determine one or more appropriatecommunication channels on which to communicate the remote activity. Forexample, master node 70 may read the annotated query execution graphassociated with the corresponding precompiled query 24 to determinewhether the corresponding precompiled query 24 calls for an index reador other suitable activity of at least one key part 30. To determine theappropriate communication channel that has access to the necessary datafor resolving the query request 54, master node 70 accesses theappropriate top level key 32 associated with corresponding precompiledquery 24 to determine the one or more key parts 30 that may be needed toresolve at least a portion of query request 54. Master node 70 may mapthe location of one or more particular key parts 30 to one or morecommunication channels using, for example, the “part_no MODnum_channels” function described above.

In this example, master node 70 accesses the appropriate top level key32, retrieves the identification of the one or more appropriate keyparts 30, maps the one or more appropriate key parts 30 to one or morecommunication channels, and communicates one or more requests for anindex read (or other suitable activity) of the one or more particularkey parts 30.

In certain embodiments, each master node 70 is capable of communicatingone or more requests to perform a particular one or more activitiesassociated with corresponding precompiled query 24 for processing by oneor more slave nodes 72 ₁-72 _(M). The requests communicated by masternodes 70 may each comprise a request package. Each request package mayinclude the one or more input variables supplied in query requests 54.For example, the one or more input variables of query request 54 mayinclude one or more of first name, last name, and social securitynumber. As another example, the one or more input variables of queryrequest 54 may include one or more of make, model, year, and color ofone or more vehicles. Each request package may also include an activityID identifying the particular activity (e.g., an index read) that themaster node 70 is requesting the one or more slave nodes 72 to perform.For example, the activity ID may direct the one or more slave nodes 72to particular locations within the DLL file. Although this descriptionfocuses primarily on embodiments, in which the requests communicated bymaster nodes 70 each comprises a request package, the present inventioncontemplates the requests comprising any suitable format, according toparticular needs.

In certain embodiments, master node 70 assigns a transaction ID to therequest package, so that when the one or more slave nodes 72 returnresults to master node 70, master node 70 will be able to match thereturned results to the appropriate request package. Each requestpackage may also include one or more of: (a) a master node identifierindicating which master node 70 sent the request package; (b) anindication of the communication channel on which master node 70communicated the respective request package; and (c) an indication ofwhether there have been any retries for the one or more activitiesassociated with the request package. In certain embodiments, the masternode identifier may include the Internet protocol (IP) address of themaster node 70, which may be globally unique. In some cases, the masternode identifier may be included as part of the transaction ID. Includingmaster node identifiers in request packages may allow the slave nodes 72that receive the request packages to know where to send the resultsobtained by the slave nodes 72. The request package may include anyother suitable information or data, according to particular needs.Certain of the information in the request package may be included in aheader portion of the request package. In certain embodiments, masternode 70 generates the request packages using one or more helper files,such as one or more DLLs, accessible to master node 70.

In certain embodiments, the request package may include a priorityidentifier for identifying a priority of the request package. Thepriority identifier may be implemented in any suitable manner, accordingto particular needs, and the priority may be based on any suitablefactors. As just one example, the priority may be a binary variable,where a value of zero indicates a low priority and a value of oneindicates a high priority, or vice versa. The priority may be auser-specified priority in query request 54. Additionally oralternatively, the priority may be based on an identification of theclient 12 that submitted the query request 54 associated with therequest package. For example, certain clients 12 may be granted orassigned a higher priority than other clients 12, so that query requests54 submitted by those clients 12 granted a higher priority tend to beprocessed more quickly by system 14 than query requests 54 submitted bythose clients 12 granted a lower priority. Additionally oralternatively, the priority may be based on the type of query request 54and/or the type of activities associated with the request package.

Each master node 70 may maintain one or more outgoing queues or othersuitable data structures 82 for storing request packages until suchrequest packages should be communicated to one or more of slave nodes 72via one or more communication channels. For example, each master node 72may maintain a separate outgoing queue 82 for each communication channelof network 80, a single outgoing queue 82 for all communication channelsof network 80, or any other suitable number of outgoing queues 82.Master nodes 70 may communicate request packages from their respectiveoutgoing queues 82 at random, in a first-in-first-out manner, or in anyother suitable manner.

In certain embodiments, master node 70 communicates the portion ofprecompiled query 24 to be processed by one or more slave nodes 72 ₁-72_(M) using a multicast signal formatted in, for example, UDP. Forexample, master node may communicate the request package to one or moreslave nodes 72 ₁-72 _(M) using a multicast signal. Master node maycommunicate the multicast signal on the one or more appropriatecommunication channels determined by reading the annotated queryexecution graph and top level key 32, as described above. For example,master node 70 communicates one or more multicast signals on one or moreappropriate communication channels, and such multicast signals may bereceived by some or all of slave nodes 72 ₁-72 ₄₀ capable of receivingsignals on the one or more communication channels on which the multicastsignal was communicated. In general, a multicast signal is received byany slave node 72 registered to receive multicast signals communicatedon a particular communication channel. As an example, master node 70communicates a request package on communication channel eleven, and oneor more slave nodes 72 having registered to receive communications senton channel eleven receive the request package.

Use of multicast signals may allow master nodes 70 to communicate asingle request to multiple slave nodes 72 registered on a single channelrather than requiring master node 70 to communicate a separate requestpackage to each of slave nodes 72 having access to the appropriate keyparts 30, which may reduce the outgoing communication bandwidth requiredfor master node 70 to communicate request packages and may reduce theprocessing required of master node 70. Furthermore, because in certainembodiments multiple slave nodes 72 have access to the same key parts30, multiple slave nodes 72 may be registered on a single communicationchannel, which may provide additional reliability in system 14.

Upon receiving a request package from a master node 70, a slave node 72may perform some initial processing on the request package. For example,each slave node may maintain one or more incoming queues or othersuitable data structures 84 for inserting incoming request packages. Incertain embodiments, the one or more incoming queues 84 may beimplemented as a circular buffer. Each slave nodes 72 may retrieverequest packages for processing from their corresponding incoming queues84 in any suitable manner according to particular needs. For example,slave nodes 72 may retrieve request packages from their respectiveincoming queues 84 at random, in a pseudo-random manner, in afirst-in-first-out manner, or in any other suitable manner.

In certain embodiments, the request packages may be associated with apriority, as described above. In such embodiments, slave nodes 72 mayconsider the priority of the request when inserting the request in theincoming queue 84 of the slave node 72. For example, the slave node 72may simply inspect the priority identifier in the request package, andif the priority identifier indicates that the request package has a highpriority, then the slave node 72 may insert the request package at thehead or substantially at the head of its incoming queue 84.

Additionally or alternatively, each slave node 72 may maintain separateincoming queues 84, one incoming queue 84 being for high priorityrequests and one incoming queue 84 being for low priority requests. Incertain embodiments, each slave node 72 may select a request forprocessing from its high priority incoming queue 84 before or morefrequently than selecting a request for processing from its low priorityincoming queue 84. Although two incoming queues 84 are described (onefor high priority requests and one for low priority requests), thepresent invention contemplates each slave node 72 maintaining a numberof incoming queues 84, each corresponding to one of a number of degreesof priority (e.g., high priority, medium priority, and low priority).Prioritizing requests received from master nodes 70 may enable may helpsystem 14 to manage throughput in processing of query requests 54received from one or more clients 12. In particular embodiments,prioritizing requests may help system 14 to increase or maximizethroughput for query requests 54 of higher importance or priority.

In embodiments in which more than one slave node 72 is registered on acommunication channel on which a master node 70 communicates a requestto perform one or more activities (e.g., in a request package), morethan one of the slave nodes 72 may receive the request communicated bythe master node 70. For example, more than one of the slave nodes 72 mayretrieve the request package from their respective incoming queues 84 toperform the one or more activities associated with the request. In somecases, it may be desirable it may be desirable to reduce or eliminatethe possibility that more than one slave node 72 will handle the requestcommunicated by the master node 70.

In certain embodiments, upon retrieving a request from queue 84 toperform the one or more activities associated with the request, slavenodes 72 are operable to communicate a notification or other suitablemessage 85 on the communication channel on which the request wasreceived. In general, notification 85 is operable to notify one or moreother slave nodes 72 on the same communication channel as the slave node72 that is sending notification 85 (and presumably the samecommunication channel on which the request was communicated by themaster node 70) that the slave node 72 that sent notification 85 ishandling the request.

Notification 85 may comprise any suitable format and include anysuitable information, according to particular needs. In certainembodiments, notification 85 includes one or more of the transaction IDassociated with the request package, one or more activity IDs includedin the request package, or any other suitable information. For example,notification 85 may include substantially similar information as wasincluded in the request package for which the slave node 72 is claimingresponsibility.

In a particular non-limiting example, ten communication channels areused and slave nodes 72 ₃ and 72 ₄₃ each have access to key part 30 ₃and are registered to receive communications sent on the thirdcommunication channel. Although particular slave nodes 72 are describedfor purposes of this example, any or all of slave nodes 72 may becapable of performing similar functionality. Suppose master node 70 ₁communicates a request package on the third communication channel (e.g.,using a multicast signal) and that the activity associated with therequest comprises an index read of key part 30 ₃. Suppose also forpurposes of this example that both slave nodes 72 ₃ and 72 ₄₃ receivethe request package and insert the request in their respective incomingqueues 84. Typically, either slave node 72 ₃ or slave node 72 ₄₃ mayfulfill this request by performing the index read of key part 30 ₃.Suppose slave node 72 ₃ retrieves the request from its incoming queue 84to perform the activity associated with the request. In this example,slave node 72 ₃ communicates a notification 85 on the thirdcommunication channel. Slave node 72 ₄₃ will receive the notification 85and will know that it does not need to fulfill the request because slavenode 72 ₃ is already doing so. In certain embodiments, slave node 72 ₄₃will remove the request from its incoming queue 85 or otherwise flag therequest as being processed by another slave node 72.

In certain embodiments, it may be possible for both slave node 72 ₃ andslave node 72 ₄₃ to begin processing the request at substantially thesame time or otherwise prior to receiving a notification 85 from theother slave node 72. In some cases, it may be desirable for one of slavenodes 72 ₃ and 72 ₄₃ to stop processing the request. In suchembodiments, both slave nodes 72 ₃ and 72 ₄₃ may send a notification onthe third communication channel and each of slave nodes 72 ₃ and 72 ₄₃may receive the notification communicated by the other slave node. Incertain embodiments, slave nodes 72 ₃ and 72 ₄₃ are able to arbitratebetween themselves (or among themselves if more than two slave nodes 72are involved) as to which will stop processing the request and whichwill continue processing the request.

For example, in certain embodiments, the arbitration may be determinedbased on a relative priority between or among the slave nodes 72registered with the same channel. The priority may be determined in anysuitable manner, according to particular needs. In certain embodiments,a relative priority is assigned to each of the slave nodes 72 operatingon the same channel. In other embodiments, the relative priority may bebased on the IP address for each slave node 72, such that a slave node72 with a higher IP address “wins” the arbitration. In yet otherembodiments, the relative priority may be calculated according to analgorithm. As just one example, the algorithm may be based in part onthe Internet protocol (IP address) assigned to each slave node 72 andone or more other suitable variables such as the activity ID associatedwith an activity in the request communicated by the master node 70. Theuse of an algorithm may be desirable because the algorithm may reduce oreliminate the chances that the same slave node 72 always orsubstantially always “wins” the arbitration.

In certain embodiments, the slave node 72 that “wins” the arbitrationwill continue handling the request communicated by the master node 70,while the one or more slave nodes 72 that “lose” the arbitration willcease processing the request. The one or more slave nodes 72 that “lose”the arbitration may also remove the request from their respectiveincoming queues 84.

Other techniques may be used to reduce or eliminate the possibility thatmore than one slave node 72 will handle a request communicated by amaster node 70. In certain embodiments, slave nodes 72 may retrieverequests in their respective incoming queues 84 in a somewhat randomfashion. For example, slave nodes 72 may randomly or pseudo-randomlyselect one of a first predetermined number of requests in theirrespective incoming queues 84 for processing. As a particular example,slave nodes 72 may select one of the first ten requests in theirrespective incoming queues 84 for processing. This may reduce thechances that more than one slave node 72 on the same communicationchannel will select the same request from their respective incomingqueues 84 to process.

Although in this example, the predetermined number is ten, thepredetermined number may be any suitable number, according to particularneeds. Additionally, although the predetermined number is described asbeing from the first positions in incoming queue 84, the predeterminednumber may be at any suitable position in the incoming queue 84. Forexample, slave nodes 72 may select one of the middle twenty entries intheir respective incoming queues 84, one of the last fifteen entries intheir respective incoming queues 84, or in any other suitable manner. Itmay be desirable, in some embodiments, to select from the firstpositions in the incoming queue 84 so that the requests are processed atleast somewhat in the order in which they were received.

Upon retrieving a request package, from incoming queue 84 for example, aslave node 72 may determine one or more activities associated with therequest package. For example, slave node 72 may determine the one ormore activities associated with the request package by reading the oneor more activity IDs specified in the request package. In thisparticular example, the received request package specifies a singleactivity ID indicating that the activity is an index read. Slave node 72may use one or more helper files to determine a portion of one or moreDLLs to access to perform the activity associated with the activity ID.Slave node may use the one or more variables specified in the requestpackage (e.g., the one or more variables provided in query request 54)to perform the activity associated with the activity ID. For example,slave node 72 may execute the portion of the one or more DLLs, using theone or more variables of the request package, to perform the activity(e.g., an index read). In one example, the one or more variables includeone or more of a first name, a last name, and a social security number.In this example, slave node 72 accesses the appropriate key part 30 toresolve the activities associated with the request package using the oneor more variables as input, which, in this example, includes retrievingone or more addresses associated with the one or more input variables.

After performing the appropriate action and retrieving or otherwiseobtaining and/or assimilating the appropriate results, slave node 72 mayinitiate return of the results for the request package to one or more ofmaster nodes 70. Slave nodes 72 may return results to appropriate masternodes 70 in any suitable manner, according to particular needs. Incertain embodiments, each slave node 72 maintains one or more outgoingqueues 86. In such embodiments, a slave node 72 may insert the retrievedresults into an outgoing queue 86 for storing such results. The resultsmay be stored as one or more result packages. For example, the number ofresult packages may depend on the quantity of results to be returned.The result packages in a single outgoing queue 86 may provide resultsfor a single request from a single master node 70, provide results fordifferent requests from the same master node 70, or provide results fordifferent requests received from different master nodes 70, according toparticular needs. In certain embodiments, each slave node 72 maymaintain a different outgoing queue 86 for results intended for eachmaster node 70; however, this is not required in all embodiments. Eachresult package may include the transaction ID included in the requestpackage to which the results correspond, so that the receiving masternode 70 can determine to which request the result package corresponds.

Typically, slave nodes 72 return results for a request received from amaster node 70 to the same master node 70 that communicated the request.Slave nodes 72 may determine the appropriate master node 70 to which tocommunicate results based on a master node identifier (which may be apart of the transaction ID of the results package) in the requestpackage communicated by the master node 70. In certain embodiments,slave nodes 72 may substantially immediately return obtained results tothe master node 70. For example, slave nodes 72 may return results asthose results are obtained (e.g., in a first-in, first-out manner).

In other embodiments, it may be desirable for master nodes 70 tosubstantially control when the slave nodes 72 may return results toappropriate master nodes 70. For example, each slave node 72 may insertits results (e.g., as result packages) into its respective outgoingqueue 86 and wait to send the results to the appropriate master node 70until the slave node 72 receives a permission-to-send message from theappropriate master node 70. In such embodiments, after obtaining resultsresponsive to a request communicated by a particular master node 70, aslave node 72 may communicate a message on a dedicated flow controlchannel, indicating that slave node 72 has results to return to theparticular master node 70. This message may be referred to as arequest-to-send message, and may be multicast, broadcast, or uni-cast.In certain embodiments, the dedicated flow control channel may be aseparate port from the other communication channels, the separate portbeing used by any or all of master nodes 70 and slave nodes 72 tocommunicate data flow control messages.

In a particular example, a transport mechanism for handling thecommunication of results from one or more slave nodes 72 to one or moremaster nodes 70 includes a high level layer and a low level transportprotocol. The high level layer may be responsible for the uniqueidentifiers associated with each communicated message (e.g., thetransaction ID associated with the message). The low-level transportprotocol may include the unique identifiers (e.g., the transaction IDassociated with the message), and one or more sequence numbersassociated with the message. The high level layer may be responsible forreassembling messages. For example, data communicated to master nodes 72may have been communicated from multiple slave nodes 72. Low-levelmessages from slave nodes 72 may that include the same unique identifierand the same low-level sequence number may be associated with the samerequest to perform one or more activities (e.g., when a requestcommunicated by a master node 70 involves multiple slave nodes 72searching multiple key parts 30). For example, either of the uniqueidentifier or the low-level sequence number associated with resultpackages communicated by slave nodes 72 may allow the receiving masternode 70 to reconstruct an appropriate order of the result packages,since the result packages may not be received at the same time orsequentially by the master node 70. The request-to-send andpermission-to-send messages may be a part of the low-level transportlayer as described below.

The master node 70 that sent the request package may, in response to therequest-to-send message, communicate a permission-to-send message to theslave node 72 indicating that the slave node 72 may communicate theresults to the master node 70. As such, slave nodes 72 may not sendresults from their outgoing queues 86 in a first-in, first-out manner,in certain embodiments, but may instead send results for which they havereceived a permission-to-send message, wherever those results may be inoutgoing queue 86.

In certain embodiments, as a master node 70 receives request-to-sendmessages from one or more slave nodes 72, it may store thoserequest-to-send messages in a queue or other suitable data structure 87.When the master node is not busy receiving data (e.g., result data fromslave nodes 72 as opposed to flow control data, in certain embodiments),the master node 70 may access queue 87, and select a request-to-sendmessage from queue 87. For example, the master node 70 may select therequest-to-send messages from queue 87 in a first-in, first-out manner.The master node 72 may, prior to sending a permission-to-send messagefor the slave node 72 associated with the selected request-to-sendmessage, determine whether the slave node 72 is available to send theresults. For example, the slave node 72 may already be busy sendingother results to another master node 70. Thus, when a master node 70receives a request-to-send message from the particular slave node 72, itmay be desirable to first determine whether the particular slave node 72is already busy sending results to another master node 70. This mayallow the master node 70 to determine, by accessing queue 87 forexample, whether any other slave nodes 72 for which the master node 70has outstanding request-to-send messages are available to send resultsto the master node 70, rather than to wait for the particular slave node72 to finish sending its certain of its results to the one or more othermaster nodes 70.

Once a master node 70 determines that a particular slave node 72 isavailable to send results to the master node 70, the master node 70 maysend a permission-to-send message to the particular slave node 72. Insome cases, if the master node 70 determines that all slave nodes 72corresponding to request-to-send messages in queue 87 of the master node70, then master node 70 may communicate the permission-to-send messageto any suitable slave node 72 corresponding to a request-to-send messagein queue 87 (e.g., the slave node 72 corresponding to the firstrequest-to-send message in queue 87).

In certain embodiments, the permission-to-send message may includeinformation regarding how much data the particular slave node 72 isallowed to send to the master node 70, which may be based on how muchbuffer or queue space is available on the master node for receivingresults from slave nodes 72. Additionally, a predefined maximum amountof data may exist, which may limit the amount of data that a master node70 can allow a slave node 72 to send to the master node 70 atsubstantially one time. Each permission-to-send message may include anyother suitable information, according to particular needs. In certainembodiments, a particular communication channel may be dedicated as aflow-control channel on which these messages may be sent, each masternode 70 being operable to listen on the flow-control channel todetermine which slave nodes 72 are currently busy.

In response to receiving a permission-to-send message, at least aportion of the result data for the master node 70 may be communicated tothe master node 70. For example, the slave node 72 may communicate theresults as a uni-cast message addressed to the particular master node70. The slave node 72 may know the identity of the particular masternode 70 based on the request package associated to which the resultscorrespond. For example, the transaction ID and/or the master nodeidentifier in the request package may be used by the slave node 72 todetermine the identity of the particular master node 70.

In certain embodiments, the total data in output queue 86 (which may ormay not be specific to the master node 70) may be broken into packages,which may be smaller than a maximum transmission unit (MTU) of thenetwork on which the data is to be sent. If there is more than oneoutput queue 86 associated with the master node 70, then data may betaken from each output queue 86 in a round-robin or other suitablemanner, and sent to the master node 70. The slave node 72 maycommunicate results as one or more result packages, which may includeone or more of the retrieved results. Each result package may include atleast the transaction ID specified in the request package to which theretrieved results correspond and any other suitable information,according to particular needs. The transaction ID may be used by themaster node 70 that receives the result package to match up the resultpackage with the appropriate request. This may assist the master nodes70 in identifying which requests are still outstanding.

In certain embodiments, once a slave node 72 has sent an allotted amountof data to the master node 70 (e.g., the amount specified in thepermission-to-send message, if the slave has that much data to send),the slave node 72 may communicate a sending-complete message. Thesending-complete message may be communicated over the flow controlchannel, so that each master node may be made aware that the slave node72 is no longer tied up sending data to the master node 70. If the slavenode 72 has additional data to send to the master node 70, the slavenode 72 may communicate another request-to-send message to the masternode 70.

In certain embodiments, each time a slave node 72 begins to sendinformation to a master node 70, the slave node 72 may send a multicastmessage indicating that the slave node 72 is busy sending data to themaster node 70. This multicast message may be received by any or all themaster nodes 70, so that the master nodes 70 can know that the slavenode 70 is busy sending data. In certain embodiments, when the slavenode 72 is finished sending data to the master node 70, the slave node72 may send another multicast signal indicating that the slave node isnow free to send data. Alternatively, each permission-to-send messagemay be associated with a predefined time limit such that after thepredefined time limit has passed, the slave node 72 to which thepermission-to-send message was sent may be automatically released by themaster node 70 and free to send messages to other master nodes 70.

In a particular example, if a slave node 72 receives apermission-to-send message from a first master node 70 and the slavenode 72 is already busy sending data to a second master node 70, theslave node 72 may terminate sending data to the second master node 70and begin sending data to the first master node 70.

It may be possible for flow control messages (e.g., request-to-sendmessages or permission-to-send messages) to be lost and fail to arriveat their intended destinations. In certain embodiments, a predefinedtime limit may be associated with each flow control message such that ifa slave node 72 does not receive a permission-to-send message inresponse to a request-to-send message within the predefined time limit,the slave node 72 may resend the request-to-send message. Furthermore,if a master node 70 does not receive a complete message in response tosending a permission-to-send message, then the master node 70 mayreinsert the request-to-send message (i.e. for which the incompleteresults were sent) into queue 87 (e.g., at the end of queue 87) to beretried at a later time. The time limit for the completed messages maybe dynamically determined based on the amount of data allotted to theslave node 72 in the permission-to-send message. For example, when amaster node 70 indicates in the permission-to-send message that a slavenode 72 may send a large amount of data, the time limit may be longerthan when a master node 70 indicates in the permission-to-send messagethat a slave node 72 may send a smaller amount of data.

In practice, master nodes 70 may each be handling a large number ofquery requests 54 received from one or more clients 12, during heavy useof system 14 for example. Additionally, a master node 70 may be handlinga query request 54 that invokes a complex precompiled query 24 thatrequires a master node 70 to communicate a large quantity of requests toone or more slave nodes 72 (e.g., via multicast signals communicated onone or more communication channels). Furthermore, a master node 70 maycommunicate a request to one or more slave nodes 72 that may require theslave nodes 72 to return a large quantity of results, as separate resultpackages for example. Thus, there may be multiple requests beingprocessed by multiple slave nodes 72 in parallel for each master node70. Furthermore, a particular request may involve multiple index readsor an index read of multiple key parts 30, which may or may not involvemultiple slave nodes 72. For these and other reasons, each master node70 may have multiple requests being processed by multiple slave nodes72, and in certain embodiments, it may be desirable for each master node70 to received results from the first available slave node 72 ratherthan waiting to receive those results in a particular order.

The above-described techniques may assist the master node 70 in managingthe return of those results, particularly in such complex situations.For example, by frequently checking queue 87, a master node 70 may keeptrack of whether the master node 70 has received a request-to-send foreach of those requests, and once the master node 70 communicates apermission-to-send to a slave node 72 and actually receives the resultsfrom the slave node 72, the master node 70 may mark the request to whichthose results are responsive of the list of outstanding requests(assuming there are no other slave nodes 72 returning results for thatrequest). In certain embodiments, a master node 70 may be operable topause the sending of requests to slave nodes 72 if the master node 70currently has too many outstanding requests for which results have notbeen returned.

In certain embodiments, slave nodes 72 may be configured to send only acertain number of results. For example, a particular slave node 72 mayreceive a request (e.g., a request package) from a master node 70,requesting that all persons an index read for all persons having thefirst name Richard. Such a request may result in a large number ofresults. Assume for purposes of this example that the particular slavenode 72 finds one thousand data entries for people having the first nameRichard. Also, assume for purposes of this example only that each slavenode 72 is configured to send no more than one hundred results at atime. Although described as a particular number of results in thisexample, each slave node 72 may be configured to only send a particularamount of data, such as ten thousand kilobytes for example.

In certain embodiments, the particular slave node 72 may communicate thefirst one hundred results and may notify the particular master node 70,in the result package that includes the first one hundred results forexample, that more results are available. The result package may alsoinclude sufficient information to allow the particular master node 70 tosend out another request for more of the found results (i.e., the nextone hundred results in the one thousand found results), and to informallow the slave node 72 that receives the next request to start wherethe previously returned results left off. This may include, for example,in indication of a location within the key part 30 for resolving therequest from the particular master node 70 where the particular slavenode 72 stopped returning results. In certain embodiments, theparticular master node 70 may send the next request for the next onehundred results to the same or a different slave node 70.

Master nodes 70 may receive returned results from slaves nodes 72, asresult packages for example. In certain embodiments, master nodes 70maintain one or more incoming queues or other suitable data structures88 for inserting retrieved results that are returned from slave nodes72. For example, each master node 70 may maintain separate incomingqueues 88 for each request that the master node 70 communicated.Alternatively, each master node may maintain a single incoming queue 88for all results. Master nodes 70 may match up the result package withthe request package based on the transaction ID specified in both theresult package and the request package. Additionally, if a master node70 receives duplicate results from one or more slave nodes 72 (e.g., iftwo slaves process the request communicated by the master node 70 andboth return the same results for the request), master node 70 may beoperable to discard one of the duplicates. Master node 70 may assimilatethe results in any suitable manner, according to particular needs.

In certain embodiments, master node 70 may need to perform additionalprocessing on the results received from one or more slave nodes 72,based on the annotated query execution graph associated with theprecompiled query 24 for which the results were retrieved. For example,master node 70 may perform one or more local activities such as a sort,a de-duplication, and/or a roll-up of the returned results. While theseactivities are provided as examples of local activities, the presentinvention contemplates these activities being remote activities,performed by one or more of slaves 72 for example. Master node 70 mayperform any other suitable processing on the returned results, accordingto particular needs. For example, the annotated query execution graphmay indicate that another request package should be sent out based onthe results received from slave nodes 72.

In certain embodiments, master node 70 receives results corresponding toa particular precompiled query 24 from more than one slave node 72operating on one or more communication channels of network 80. In suchembodiments, master node 70 may assimilate the results received fromeach of the slave nodes 72 responsive to the same query request 54. Forexample, master node 70 may assimilate the results by using thetransaction ID in the return packages for the results to determine whichresults are responsive to the same query request 54.

Master nodes 70 may initiate communication of the results to the client12 that submitted the query request corresponding to the results. Theresults may be communicated to client 12 in any suitable manner,according to particular needs. For example, the results may becommunicated to client 12 in a manner that bypasses query receivingmodule 60. The results may comprise any suitable format, according toparticular needs, such as XML or HTML. The results may be communicatedto client 12 in a single communication or in multiple communications.These results may be displayed on client 12 (e.g., on GUI 52 of clientsystem 12) in any suitable manner, according to particular needs.

FIG. 3 illustrates an example query execution graph 100 associated withan example precompiled query 24. FIG. 3 illustrates just one exampleembodiment of a query execution graph. It should be appreciated thatother embodiments of a query execution graph may be used withoutdeparting from the scope of the present invention. In this example,query execution graph 100 includes five activities 102-110. Althoughquery execution graph 100 includes five activities 102-110 in thisexample, query execution graph 100 could include any other number ofactivities without departing from the scope of the present invention.

In certain embodiments, query execution graph 100 includes activities102-106, each capable of providing one or more desired responses orranges of responses upon receiving a set of input variables (e.g.,specified in a query request 54). Activities 102-106 may include, forexample, remote activities, local activities, and/or a combination ofremote and local activities. As used throughout this document, the term“remote activity” or “remote activities” refers to a particular activitywithin a precompiled query that calls for the execution of at least aportion of that activity on one or more slave nodes. A “local activity”is a particular activity within a precompiled query that is executed onthe master node processing the precompiled query.

In this particular embodiment, each of activities 102-106 includes oneor more remote activities, such as, for example, one or more indexreads, one or more record reads, one or more aggregations, or any otheractivity that necessitates the use of one or more slave nodes. Moreover,each of activities 102-106 has access to one or more associated DLLsand/or helper files capable of assisting a master node 70 or slave node72 in processing the one or more remote activities associated withactivities 102-106.

In this example, activities 108-110 include one or more localactivities, such as, for example, one or more sorts, one or morede-duplications, one or more roll ups, or any other activity capable ofbeing performed on the master node that received the request (e.g.,query request 54). Each of activities 108-110 has access to one or moreassociated DLLs and/or helper files capable of assisting a master node70 in processing the one or more local activities associated withactivities 108-110.

In one non-limiting example, query execution graph 100 illustrates aprecompiled query capable of returning one or more desired addresses orrange of addresses when any combination of one or more first names, oneor more last names, or one or more social security numbers are providedby a user of a database system, such as system 14 of FIGS. 1 and 2(e.g., as part of a query request 54). In that example, activity 102 iscapable of returning one or more desired addresses when a user inputs afirst name and last name, while activity 104 is capable of returning oneor more desired addresses when a user inputs a range social securitynumber. Moreover, activity 106 is capable of returning one or moredesired addresses when a user inputs one or more last names.

In that example, activity 108 operates to determine which inputs havebeen provided by the user and to select an appropriate one of activities102-106 to resolve the user's request. In various embodiments, activity108 includes or has access to logic that enables activity 108 todetermine the best activity 102-106 to use in resolving the user'srequest. In some cases, activity 108 can include logic that determinesthe probability of each activity 102-106 returning the desired addressbased on the inputs and selects the activity with the highestprobability. For example, the logic could indicate that when a socialsecurity number and a last name are provided, implementing activity 104is most likely to return the desired address. In this example, activity110 operates to provide an output signal that includes the desiredaddress to the user.

FIG. 4 illustrates an example sorted table 200 that includes a pluralityof key parts 224. FIG. 4 illustrates just one example embodiment of asorted table. It should be appreciated that other embodiments of asorted table may be used without departing from the scope of the presentinvention. In this example, sorted table 200 includes four fields202-208. Although sorted table 200 include four fields 202-208 in thisexample, sorted table 200 could include any other number of fieldswithout departing from the scope of the present disclosure.

In one non-limiting example, sorted table 200 is sorted by first field202, which includes a name first name for a plurality of data entries.Sorted table 200 is then sorted by second field 204, which includes alast name that is associated with the first name of the respective dataentry. Finally, sorted table 200 is sorted by third field 206, whichincludes an address associated with the respective entry. Fourth field208 may include any data entry identifier 208, such as, for example, theposition of the data entry within the sorted database, a pointer to thelocation of additional data, or any other appropriate data entryidentifier. In other embodiments, third field 206 may include a pointerto the location of the desired address that is stored in another table.

Fields 202-206 of sorted table 200 may be populated with data generatedby and/or stored on a database system, such as second database system 40of FIG. 1. In this particular example, fields 202-206 are populated withdata sorted on a database system after receiving a request to generate asorted table for use in resolving at least a portion of a precompiledquery. In that example, the precompiled query is capable of returningone or more desired addresses when a first name and last name areprovided to a database system, such as first database system 14 of FIGS.1 and 2.

In most cases, the one or more sorts of data result in sorted table 200being distributed over one or more of particular example, first key part234 _(a) includes the addresses for names ranging from Aaron Aaskey toChad Thomas, second key part 234 _(b) includes the addresses for thenames ranging from Chad Tomas to Connor McConnell, and last key part 234_(N) includes addresses for the names Yen Lee to Zack Zymol.

In this particular example, after key part 234 _(a)-234 _(N) arepopulated with data generated by and/or stored on a database, each ofkey parts 234 _(a)-234 _(N) is communicated to another database system,such as database system 14 of FIGS. 1 and 2, for use in resolving arequest to execute a precompiled query. In some cases, a particularprecompiled query can use sorted table 200 to identify one or moreaddresses for all persons having the first name Chad and that have alast name that is phonetically similar to Thomas. In that case, theparticular precompiled query uses key parts 234 a and 234 b to identifythe addresses of at least Chad Thomas and Chad Tomas.

FIG. 5 illustrates an example top level key 324 associated with a sortedtable. FIG. 5 illustrates just one example embodiment of a top level keyassociated with a sorted table, such as table 200 of FIG. 4. It shouldbe appreciated that other embodiments of a top level key may be usedwithout departing from the scope of the present disclosure. In thisexample, top level key 324 includes four fields 202-208. Although toplevel key 324 includes four fields 202-208 in this example, top levelkey 324 could include any other number of fields without departing fromthe scope of the present disclosure.

In one non-limiting example, a database system, such as second databasesystem 40 of FIG. 1, generates top level key 324. Top level key 324includes a first field 202, which includes a first name for a pluralityof data entries. Second field 204 includes a last name that isassociated with the first name of the respective data entry; while thirdfield 206 includes an address associated with the respective entry.Fourth field 208 can include any key part location identifier, such as,for example, the identification of the respective node, such as node 44of FIG. 1, that stores the key part.

In this example, top level key 324 operates to identify the locationwithin a database system of each key part associated with a sortedtable. In this particular example, fields 202-206 are populated withdata from the sorted table for use with a particular precompiled queryof a database system, such as first database system 14 of FIGS. 1 and 2.In that example, the particular precompiled query is capable ofreturning one or more desired address when one or more first names andone or more last names are provided to the a database system.

In this example, fields 202-206 of top level key 324 identify the lastdata entry of a particular key part, such as key parts 234 _(a)-234 _(n)of FIG. 4, associated with a sorted table. Fourth field 208 identifiesthe location of the last data entry for the particular key part, such asa particular second node 44 of FIG. 1. For example, top level key 324can be used to identify that all the names following Chad Thomas up toand including Connor McConnell are located on node 44 ₂ of seconddatabase system 40 of FIG. 1. Consequently, top level key 324 identifiesthe location of each data entry within a sorted table.

In this particular example, after top level key 324 is created, toplevel key 324 is communicated to another database system, such asdatabase system 14 of FIGS. 1 and 2, for use in resolving a request toexecute a precompiled query. In some cases, a particular precompiledquery can use top level key 324 to identify the location of one or morekey parts within a database system. In this example, the database systemoperates to map the location of each key part from its respective secondnode 44 to one or more communication channels associated with thedatabase system. In some cases, the database system may include or haveaccess to one or more functions capable of mapping the location of thekey parts to one or more channel numbers associated with the databasesystem, such as, for example, the “part_no MOD num_channels” functiondescribed above.

FIG. 6 illustrates an example method for processing one or more queryrequests 54 in accordance with one embodiment of the present invention.The method described with reference to FIG. 6 illustrates just oneexample method of processing queries in certain embodiments of thepresent invention. Moreover, in describing the method illustrated inFIG. 6, various example query requests 54 are described, which are forexample purposes only and should not be used to limit the scope of thepresent invention.

Furthermore, for purposes of this example, it is assumed that firstdatabase system 14 includes one or more precompiled queries 24 ₁-24_(W), one or more top level keys 32 ₁-32 _(X), and a plurality of keyparts 30 ₁-30 _(N) necessary to resolve precompiled queries 24 ₁-24 _(W)deployed to nodes 20 ₁-20 _(M). In this particular embodiment, system 14stores each of precompiled queries 24 ₁-24 _(W) on each of master nodes70 ₁-70 _(M) and slave nodes 72 ₁-72 _(M). First database system 14 alsoexecutes one or more of precompiled queries 24 ₁-24 _(W) upon receivinga query request to execute a particular precompiled query 24 from a userof system 14. In some cases, the query request 54 to execute aparticular precompiled query 54 can be received by first database system14 from a client, such as client 12 of FIGS. 1 and 2. In variousembodiments, a user can request execution of a particular precompiledquery 54 by connecting to system 14 through any appropriate means, suchas through a host coupled to system 14.

At step 400, system 14 receives a query request 54 from client 12. Forexample, a user of client system 12 may submit a query request 54, whichmay identify one or more input variables and a precompiled query 24corresponding to the query request 54. For example, a user may desire tosearch for one or more addresses when any combination of first name,last name, and social security number are provided in query request 54to system 10. Assuming that such a query is a precompiled query 24, theuser may provide one or more of first name, last name, and socialsecurity number, along with an indication of the precompiled query 24corresponding to the query request 54. Alternatively, if no precompiledquery 24 corresponds to the query request 54, query request may identifyone or more input variables and a desired output of the query request 54without identifying a particular precompiled query 24 associated withthe query request 54.

In one particular example, system 14 receives a plurality of queryrequests 54 from one or more users of system 14 to execute precompiledquery 24 ₂. In that example, precompiled query 24 ₂ operates to returnone or more desired addresses when any combination of one or more firstnames, one or more last names, and one or more social security numbersare provided in query request 54 to system 14.

In one particular non-limiting example, a first user of system 14provides a first query request 54 that seeks to have precompiled query24 ₂ return one or more desired addresses for all persons named ChadThomas. A second user of system 14 provides a second query request 54that seeks to have precompiled query 24 ₂ return a plurality ofaddresses for all the social security numbers within a particular rangeof numbers (e.g., from 111-22-3333 to 111-22-3417).

At step 402, one or more master nodes 70 are selected to receive theparticular precompiled query 24 that resolves or otherwise correspondsto the user's query request 54. In most cases, system 14 selects onlyone of master nodes 70 ₁-70 _(M) to receive query request 54 from theuser and to execute the particular precompiled query 24 for resolvingthe user's query request 54. In certain embodiments, query receivingmodule 60 receives query request from client 12 via network 14. Forexample, in embodiments in which query receiving module 60 includes loadbalancing functionality, query receiving module 60 may initiate routingof query request 54 to one or more particular master nodes 70 based onany suitable load balancing technique for managing the load of queryrequests 54 that each master node 70 is handling. As described abovewith reference to FIG. 1, first database system 14 may be coupled toquery receiving module 60 via a link 66, and query receiving module 60may be separate from database system 14. Alternatively, query receivingmodule 60 may a part of system 14. In certain embodiments in which queryreceiving module 60 is separate from system 14, query receiving module60 may receive query request 54 prior to system 14 receiving queryrequest 54.

Additionally or alternatively, in certain embodiments, particularclients 12 may be pre-assigned to particular master nodes 70, and queryrequests 54 received from a client 12 may be routed to a correspondingpre-assigned master node 70 for the client 12. In yet other embodiments,query receiving module 60 may select which master node 70 will receivequery requests 54 in a round-robin fashion. In yet other embodiments,query receiving module may select which master node 70 will receivequery requests 54 in a random or pseudo-random fashion.

In one particular non-limiting example, master node 70 ₃ is selected toreceive and process the first query request 54 received from the firstuser, and master node 70 ₄ is selected to receive and process the secondquery request 54 received from the second user. Although master nodes 70₃ and 70 ₄ are selected to receive and process the first and secondquery requests 54 in this example, any of master nodes 70 ₁-70 _(M)could receive and process query requests 54 without departing from thescope of the present disclosure.

At step 404, master nodes 70 ₂ and 70 ₃ may process at least a portionof first and second query requests 54, respectively, to determine which,if any, of precompiled queries 34 ₁-34 _(W) corresponds to first andsecond query requests 54. For example, query requests 54 may eachinclude an indication their corresponding precompiled queries 24 (i.e.,precompiled query 24 ₂ in this example). If a query request 54 does notcorrespond to one of precompiled queries 24 ₁-24 _(W), master node 70may, in certain embodiments, initiate dynamic creation of one or morekeys parts 42 for resolving query 22. In this particular example, masternodes 70 ₂ and 70 ₃ determine that both first query request 54 andsecond query request 54 are requests to execute precompiled query 24 ₂.

At step 406, each of master nodes 70 ₂ and 70 ₃ reads the annotatedquery execution graph of precompiled query 24 ₂. Based on the reading ofthe annotated query execution graph of precompiled query 24 ₂, masternodes 70 ₂ and 70 ₃ determine whether there are any local activitiesassociated with precompiled query₂ to be performed by master nodes 70 ₂and 70 ₃, respectively, at step 408. For example, master nodes 70 ₂ and70 ₃ may determine whether one or more local activities are to beperformed prior to initiating performance of any remote activities. Insome cases, master nodes 70 ₂ and 70 ₃ determine whether the annotatedquery execution graph calls for a local activity by the uniqueidentification assigned to each activity (e.g., the activity IDs) forprecompiled queries 24 ₂ and 24 ₃. In some embodiments, master nodes 202₃ and 202 ₄ determine that precompiled query 118 ₂ can be fully executedon master nodes 202 ₃ and 202 ₄. If master nodes 70 ₂ and 70 ₃ determinethat there are local activities to be performed, master nodes 70 ₂ and70 ₃ perform those activities at step 410.

At step 412, master nodes 70 ₂ and 70 ₃ determine whether there are anyremote activities (i.e. an activity for performance on one or more slavenodes 72) associated with precompiled query 24 ₂ to be performed. Insome cases, master nodes 70 ₂ and 70 ₃ determine whether the annotatedquery execution graph calls for a remote activity by the uniqueidentification assigned to each activity (e.g., the activity IDs) forprecompiled queries 24 ₂ and 24 ₃. If master nodes 70 ₂ and 70 ₃determine that there are one or more, the method proceeds to step 414,described below. If master nodes 70 ₂ and 70 ₃ determine that there areno remote activities to be performed, then the method proceeds to step432, described below.

In this particular embodiment, master nodes 70 ₂ and 70 ₃ determine thatprecompiled queries 24 ₂ calls for one or more remote activities and theinteraction of one or more of slave nodes 72 ₁-72 _(M). The remoteactivities may include, for example, an index read, a record read, anaggregation, or any other activity that calls for the use of one or moreslave nodes 72. In this example, each of master nodes 70 ₂ and 70 ₃reads the annotated query execution graph associated with precompiledquery 24 ₂ and determines that precompiled query 24 ₂ calls for an indexread of at least one key part 30. At step 414, master nodes 70 ₂ and 70₃ determine the particular remote activities called for by the annotatedquery execution graph of precompiled query 24 ₂. In this particularexample, master node 202 ₃ determines that an activity associated withresolving the first and second query requests 54 calls for an indexread. Although the remote activity comprises an index read in thisparticular example, the present invention contemplates the remoteactivity comprising any suitable activities according to particularneeds, and the subsequent steps of the method being modified in anysuitable manner to handle those other remote activities.

At step 416, master nodes 70 ₂ and 70 ₃ determine one or more key parts70 for resolving their respective query requests 54. For example, masternodes 70 ₂ and 70 ₃ may access the top level key associated withprecompiled query 24 ₂ to determine the one or more key parts 70 forresolving their respective query requests 54. In this particularexample, master node 70 ₂ determines that first query request 54 callsfor an index read for all persons having the first name Chad and anindex read for all the Chad's that have a last name that is phoneticallysimilar to Thomas. Consequently, in this example, master node 70 ₂determines that precompiled query 24 ₂ calls for index reads of keyparts 30 ₁₇ and 30 ₁₈ to resolve the first query request 54. Meanwhile,master node 70 ₃ determines that addresses associated with socialsecurity numbers between 111-22-3333 and 111-22-3385 are located in keypart 30 ₂₀ and that addresses associated with social security numbersbetween 111-22-3386 and 111-22-3417 are located on key part 30 ₂₁. Thus,master node 70 ₃ determines that precompiled query 24 ₂ calls for indexreads of key parts 30 ₂₀ and 30 ₂₁ to resolve the second query request54.

At step 418, master nodes 70 ₂ and 70 ₃ determine one or morecommunication channels on which to send a request to perform the remoteactivity determined at step 414. In certain embodiments, master nodes 70₂ and 70 ₃ may map the one or more key parts determined at step 416 toone or more communication channels, using, for example, the “part_no MODnum_channels” function described above. In this particular example, todetermine the appropriate communication channel for key parts 30 ₁₇, 30₁₈, 30 ₂₀, and 30 ₃, each of master nodes 70 ₂ and 70 ₃ performs the“part_no MOD num_channels” function described above. Each of masternodes 70 ₂ and 70 ₃ identifies the appropriate communication channels ofthe respective key parts 30. In this example, master node 70 ₂ maps thelocation of key parts 30 ₁₇ and 30 ₁₈ to communication channelsseventeen and eighteen, respectively. Master node 70 ₃ maps the locationof key parts 30 ₂₀ and 30 ₂₁ to communication channels twenty andtwenty-one, respectively.

At step 420, each of master nodes 70 ₂ and 70 ₃ may create one or morerequest packages for communication on their respective determinedcommunication channels. The request packages may include any suitableinformation or data, as described above with reference to FIG. 2.Typically, a request package will include at least an identification ofa requested activity for at least one slave node 72 to perform, the oneor more input variables or other parameters provided in query request54, and a transaction ID identifying the particular instance of therequest. Although formation and communication of request packages aredescribed in this example, the present invention contemplates masternodes 70 requesting one or more slave nodes 72 to perform activities(e.g., index reads) in any suitable manner, according to particularneeds.

At step 422, master nodes 70 ₂ and 70 ₃ communicate their respectivelycreated request package via the determined one or more communicationchannels. In this example, master node 70 ₂ communicates a requestpackage that includes a request for one or more index reads of key parts30 ₁₇ and 30 ₁₈ on each of communication channels seventeen andeighteen. Meanwhile, master node 70 ₃ communicates a request packagethat includes a request for one or more index reads of key parts 30 ₂₀and 30 ₂₁ on each of communication channels twenty and twenty-one. Inthis particular embodiment, each of master nodes 70 ₂ and 70 ₃communicates the request packages using a multicast signal format. Eachof master nodes 70 ₂ and 70 ₃ communicates the multicast signals to allof slave nodes 72 ₁-72 _(M) that are capable of receiving signals on theappropriate communication channels (e.g., to all of slave nodes 72 ₁-72_(M) that have registered to receive signals on the appropriatecommunication channels).

At step 424, one or more slave nodes 72 that are capable of receivingsignals on the appropriate one or more communication channels on whichmaster nodes 70 ₂ and 70 ₃ communicated the request packages receivesthe request package. In certain embodiments, all of the slave nodes 72capable of receiving signals on the appropriate one or morecommunication channels on which master nodes 70 ₂ and 70 ₃ communicatedthe request packages receive the request package.

In this example, slave nodes 72 ₆ and 72 ₁₆ operate to store and/orprovide access to key part 30 ₁₇, and slave nodes 72 ₂ and 72 ₁₆ operateto store and/or provide access to key part 30 ₁₈. Moreover, slave nodes72 ₆ and 72 ₁₆ are capable of receiving requests on communicationchannel seventeen, and slave nodes 72 ₂ and 72 ₁₆ are capable ofreceiving requests on communication channel eighteen. In this particularexample, master node 70 ₃ communicates one or more requests in one ormore multicast signals on communication channel seventeen to each ofslave nodes 72 ₆ and 72 ₁₆ and on communication channel eighteen to eachof slave nodes 72 ₂ and 72 ₁₆.

Also in this example, slave nodes 72 ₁ and 72 ₁₁ operate to store and/orprovide access to key part 30 ₂₀, and slave nodes 72 ₂₂ and 72 ₃₄operate to store and/or provide access to key part 30 ₂₁. Moreover,slave nodes 72 ₁ and 72 ₁₁ are capable of receiving requests oncommunication channel twenty, and slave nodes 72 ₂₂ and 72 ₃₄ arecapable of receiving requests on communication channel twenty-one. Inthis particular example, master node 70 ₄ communicates a request in amulticast signal on communication channel twenty to each of slave nodes72 ₁ and 72 ₁₁ and another request in a multicast signal oncommunication channel twenty-one to each of slave nodes 72 ₂₂ and 72 ₃₄.

At step 426, one or more of the slave nodes 72 that received the requestpackage at step 424 processes the request package. For example, one ormore of the slave nodes 72 that received the request package may performthe one or more activities associated with the request package. Incertain embodiments, the one or more activity IDs included in therequest package may direct the one or more slave nodes 72 to particularlocations within an associated DLL file. The one or more slave nodes 72may execute the relevant portions of the DLL files to perform the one ormore activities requested in the request package. In this example, theactivity comprises an index read, and one or more of the receiving slavenodes 72 may perform the index read to resolve at least a portion ofquery request 54. In certain embodiments, it may be possible for morethan one of the receiving slave nodes 72 to begin processing the requestpackage and performing the one or more activities identified in therequest package. Any suitable mechanism or technique may be used toresolve this duplicative processing if desired or appropriate. Anexample method for reducing or eliminating the possibility that morethan one slave node 72 will handle the request communicated by a masternode 70 is described below with reference to FIG. 7.

In this particular example, at least one of slave nodes 72 ₆ and 72 ₁₆processes the multicast signal (e.g., the request package) communicatedon communication channel seventeen and determines the addresses for allpersons having the first name Chad and a last name phonetically similarto Thomas. Additionally, at least one of slave nodes 72 ₁ and 72 ₁₁processes the request communicated on communication channel twenty anddetermines the addresses for all persons having a social security numberbetween 111-22-3333 and 111-22-3385 to master node 70 ₃

At step 428, at least one of the slave nodes 72 that processed therequest package returns one or more of the results to a master node 70.In certain embodiments, slave nodes 72 may return results to the samemaster node 70 that communicated the request package. Slave nodes 70 mayreturn results as a result package, which may include any suitableinformation, according to particular needs. Typically, a result packageincludes at least the transaction ID included in the request package andone or more results. In certain embodiments, slave nodes 72 maycommunicate all retrieved results to master nodes 70 as a single resultpackage. In alternative embodiments, slave nodes 72 may communicate theretrieved results as multiple result packages to master nodes 70. Forexample, if the quantity of retrieved results exceeds a predeterminedsize, slave nodes 72 may communicate the retrieved results as multiplereturn packages to master nodes 70.

In this particular example, at least one of slave nodes 72 ₆ and 72 ₁₆returns the addresses for all persons having the first name Chad and alast name phonetically similar to Thomas to master node 70 ₂. Inaddition, at least one of slave nodes 72 ₂ and 72 ₁₆ processes themulticast signal communicated on communication channel eighteen andreturns all persons having the first name Chad and a last namephonetically similar to Thomas to master node 70 ₂. In addition, atleast one of slave nodes 72 ₁ and 72 ₁₁ processes the requestcommunicated on communication channel twenty and returns the addressesfor all persons having a social security number between 111-22-3333 and111-22-3385 to master node 70 ₃. In addition, at least one of slavenodes 72 ₂₂ and 72 ₃₄ processes the request communicated oncommunication channel twenty-one and returns all persons having a socialsecurity number between 111-22-3386 and 111-22-3417 to master node 70 ₃.

At step 430, master nodes 70 ₂ and 70 ₃ receive the results communicatedby the respective slave nodes. The method then returns to step 408,where the master nodes again read the annotated query execution graphfor precompiled query 24 ₂ to determine if there are any localactivities to be performed. The local activities may include anysuitable activities, according to particular needs. For example, theannotated query execution graph may indicate that the receiving masternode 70 should perform a sort, an aggregation, a de-duplication, or anyother suitable activity on the returned results.

If master nodes 70 ₂ and 70 ₃ determine at step 408 that one or morelocal activities are to be performed, master nodes 70 ₂ and 70 ₃ performthose activities at step 410. The method then proceeds to step 412,where master nodes 70 ₂ and 70 ₃ determine whether any additional remoteactivities should be performed. For example, depending on the annotatedquery execution graph for precompiled query 24 ₂, an additional indexread or other suitable remote activity may be needed. If it isdetermined that one or more additional remote activities are to beperformed, master nodes 70 ₂ and 70 ₃ initiate performance of thoseadditional remote activities. If master nodes 70 ₂ and 70 ₃ determinethat no additional remote activities are to be performed, the methodproceeds to step 432 where master nodes 70 ₂ and 70 ₃ determine whetherany additional processing is to be performed. For example, theadditional processing may include formatting the results of queryrequest 54 before returning those results to the user. If additionalprocessing is to be performed, master nodes 70 ₂ and 70 ₃ perform theadditional processing at step 434. If no additional processing is to beperformed, or once the additional processing has been performed at step434, master nodes 70 ₂ and 70 ₃ may communicated the results of queryrequest 54 to client 12. For example, master nodes 70 ₂ and 70 ₃ mayreturn the results to the requesting clients 12 for display on GUI 52 ofeach client 12. The results may be communicated to the requestingclients 12 in a single or multiple communications, according toparticular needs. The results communicated to the requesting clients 12in any suitable format, according to particular needs, such as XML orHTML.

Although a particular method for processing a query request 54 has beendescribed with reference to FIG. 6, the present invention contemplatesany suitable method for processing a query request in accordance withthe present disclosure. Thus, certain of the steps described withreference to FIG. 6 may take place simultaneously and/or in differentorders than as shown. Moreover, system 10 may use methods withadditional steps, fewer steps, and/or different steps, so long as themethods remain appropriate. Additionally, certain steps may be repeatedas needed or desired.

FIG. 7 illustrates an example method for processing a request to performan activity associated with a precompiled query 24 communicated on acommunication channel by a master node 70 and received on thecommunication channel by two or more slave nodes 72. In certainembodiments, the method described with reference to FIG. 7 may reduce oreliminate the possibility that more than one slave node 72 will handlethe request communicated by the master node 70. Although a particularcommunication channel is described with reference to this example, thisis for example purposes only and any suitable communication channel maybe used. Additionally, although each slave node 72 is described asprocessing a single request at a time, in certain embodiments, eachslave node may include a number of threads each operable to processrequests received from master nodes 70 substantially simultaneously.Furthermore, although two slave node 72 are described for purposes ofthis example, any suitable number of slave nodes 72 may be operable toreceive requests communicated on the communication and to performcertain steps of the method, according to particular configurations ofsystem 14. Moreover, although the request communicated by the masternode 70 is described as including a single activity, the request mayinclude any suitable number of activities, according to particularneeds.

At step 500, a master node 70 communicates on a communication channel arequest to perform an activity associated with a precompiled query 24.In certain embodiments, the request may comprise a request package asdescribed above with reference to FIG. 2. The request may becommunicated as a multicast signal or in any other suitable manner overthe communication channel.

At step 502, a first slave node 72 receives the request communicated bythe master node 70. For example, the first slave node 72 may beregistered to receive multicast communications over the communicationchannel. In certain embodiments, the request communicated by the masternode 70 may include a request to perform and index read or othersuitable activity with respect to a particular one or more key parts 70.In such embodiments, the first slave node 72 may store or otherwise haveaccess to such one or more key parts 70. At step 504, the first slavenode 72 may insert the received request in incoming queue 84 of firstslave node 72. Although described as a queue, incoming queue 84 mayinclude an suitable data structure, according to particular needs (e.g.,a circular buffer).

At step 506, a second slave node 72 receives the request communicated bythe master node 70. For example, the second slave node 72 may beregistered to receive multicast communications over the communicationchannel. In certain embodiments, the request communicated by the masternode 70 may include a request to perform and index read or othersuitable activity with respect to a particular one or more key parts 70.In such embodiments, the second slave node 72 may store or otherwisehave access to such one or more key parts 70. At step 508, the secondslave node 72 may insert the received request in incoming queue 84 ofsecond slave node 72. Although described as a queue, incoming queue 84may include an suitable data structure, according to particular needs(e.g., a circular buffer).

In certain embodiments, steps 502 through 508 may occur at substantiallythe same time. Furthermore, although first and second slave nodes 72 aredescribed, the present invention contemplates any suitable number ofslave nodes 72 receiving the request communicated by the master node 70on the communication channel. As just one example, any suitable numberof slave nodes 72 may be registered to receive multicast signalscommunicated on the communication channel.

At step 510, the first slave node 72 selects substantially at random oneof a first predetermined number of requests in its respective incomingqueue 84 to handle. As a particular example, first slave node 72 mayselect one of the first ten requests in its incoming queue 84 forprocessing. This may reduce the chances that more than one slave node 72on the same communication channel (e.g., the second slave node 72) willselect the same request from their respective incoming queues 84 toprocess. Alternatively, the first slave node 72 may process incomingqueue 84 in a standard first-in, first-out fashion or in any othersuitable manner, according to particular needs. In some cases, therequest received from the master node 70 may be the only request in theincoming queue 84 of the first slave node 72 (e.g., when database system14 is less busy). Alternatively, first slave node 72 may include anumber of threads each processing requests received from one or more ofmaster nodes 70, at least one of which may not be currently busyprocessing a request. In such cases, the first slave node 72 maysubstantially immediately begin handling the request.

At step 512, the first slave node 72 may send a notification 85 over thecommunication channel, indicating that the first slave node 72 ishandling the retrieved request, and begin processing or otherwisehandling the request. The notification 85 communicated by the firstslave node 72 may include any suitable information, according toparticular needs. In certain embodiments, the notification 85 includesone or more of an identification of the first slave node 72, atransaction ID associated with the request, an activity ID associatedwith the activity in the request, or any other suitable information,according to particular needs.

At step 514, the second slave node 72 selects one of a firstpredetermined number of requests in its respective incoming queue 84 tohandle substantially at random. As a particular example, the secondslave node 72 may select one of the first ten requests in its incomingqueue 84 for processing. This may reduce the chances that more than oneslave node 72 on the same communication channel (e.g., the first slavenode 72) will select the same request from their respective incomingqueues 84 to process. Alternatively, the second slave node 72 mayprocess incoming queue 84 in a standard first-in, first-out fashion orin any other suitable manner, according to particular needs. In somecases, the request received from the master node 70 may be the onlyrequest in the incoming queue 84 of the second slave node 72 (e.g., whendatabase system 14 is less busy). Alternatively, second slave node 72may include a number of threads each processing requests received fromone or more of master nodes 70, at least one of which may not becurrently busy processing a request. In such cases, the second slavenode 72 may substantially immediately begin handling the request.

At step 516, the second slave node 72 may send a notification 85 overthe communication channel, indicating that the second slave node 72 ishandling the retrieved request, and begin processing or otherwisehandling the request. The notification 85 communicated by the secondslave node 72 may include any suitable information, according toparticular needs. In certain embodiments, the notification 85 includesone or more of an identification of the second slave node 72, atransaction ID associated with the request, an activity ID associatedwith the activity in the request, or any other suitable information,according to particular needs.

At step 518, the first slave node 72 may receive the notification sentby the second slave node 72, indicating that the second slave node 72 isprocessing the request. At step 520, the second slave node 72 mayreceive the notification 85 communicated by first slave node 72,indicating that the second slave node 72 is processing the request.

At step 522, the first slave node 72 determines whether the notification85 received from the second slave node 72 indicates that the secondslave node 72 has selected and begun processing the same request thatthe first node has selected and begun processing. If the first slavenode 72 determines that the second slave node is not processing the samerequest that the first slave node 72 has already begun processing, atstep 524, each of the first and second slave nodes 72 continuesprocessing their respective selected requests.

If, at step 522, the first slave node 72 determines that the secondslave node 72 is processing the same request that the first slave node72 is processing, the first slave node 72 may initiate arbitrationbetween the first and second slave nodes 72, at step 526, to determinewhich of the first and second slave nodes 72 will stop processing therequest and which will continue processing the request. Although in thisexample the first slave node 72 is described as initiating arbitration,either or both of the first and second slave nodes 72 may initiatearbitration in this example.

In certain embodiments, for example, the arbitration may be determinedbased on a relative priority between the first and second slave nodes72. The priority may be determined in any suitable manner, according toparticular needs. In certain embodiments, a relative priority isassigned to each of the first and second slave nodes 72. In otherembodiments, the relative priority may be based on the Internet protocol(IP) address for each of the first and second slave nodes 72, such thata slave node 72 with a higher IP address “wins” the arbitration. In yetother embodiments, the relative priority may be calculated according toan algorithm. As just one example, the algorithm may be based in part onthe Internet protocol (IP address) assigned to each of the first andsecond slave nodes 72 and one or more other suitable variables such asthe activity ID associated with an activity in the request communicatedby the master node 70. The use of an algorithm may be desirable becausethe algorithm may reduce or eliminate the chances that the same slavenode 72 always or substantially always “wins” the arbitration.

At step 528, a determination is made as to which of the first or secondslave nodes 72 “won” the arbitration. If the first node “wins” thearbitration, the first slave node 72 may continue to process the requestand the second slave node 72 may cease processing the request at step530. If the first slave node 72 “loses” the arbitration at step 528,then the first slave node 72 may cease processing the request and thesecond slave node 72 may continue processing the request at step 532.

Although a particular method for performing an activity associated witha precompiled query 24 has been described with reference to FIG. 7, thepresent invention contemplates any suitable method for performing anactivity associated with a precompiled query in accordance with thepresent disclosure. Thus, certain of the steps described with referenceto FIG. 7 may take place simultaneously and/or in different orders thanas shown. Moreover, system 10 may use methods with additional steps,fewer steps, and/or different steps, so long as the methods remainappropriate. Additionally, certain steps may be repeated as needed ordesired.

FIG. 8 illustrates an example method for managing the receipt andprocessing of query requests 54 at one or more master nodes 70 of thedatabase system. In certain embodiments, the method described withreference to FIG. 8 may help system 14 to manage throughput inprocessing of query requests 54 received from one or more clients 12.Although particular master nodes 70 are described with reference to FIG.8, any or all of the master nodes 70 of system 14 may be capable ofperforming the method without departing from the scope of the presentinvention. Additionally, although two clients 12 are described as beingassociated with a particular master node 70, the present inventioncontemplates any suitable number of clients 12 being associated with theparticular master node 70. Furthermore, it will be assumed, for purposesof this example, that each master node 70 of system 14 is operable toreceive and process a predetermined number of query requests 54substantially concurrently. For example, each master node 70 may includea particular number of threads, each operable to receive and process adifferent query request 54 substantially concurrently.

At step 600, a first subset of the predetermined number of queryrequests 54 that a particular master node 70 may receive and processsubstantially concurrently may be assigned to a first client 12. Thefirst subset may be defined as a number of query requests 54 that theparticular master node 70 may process for the first client 12 atsubstantially the same time. For example, the particular master node 70may include a particular number of threads (e.g., thirty), each operableto receive and process a different query request 54 substantiallyconcurrently. A predetermined number of those threads may be assigned tothe first client 12 for receiving and processing query requests 54received from the first client 12. In this example, twenty of thepredetermined number of query requests 54 are assigned to first client12 as the first subset.

At step 602, a second subset of the predetermined number of queryrequests 54 that the particular master node 70 may receive and processsubstantially concurrently may be assigned to a second client 12. Thesecond subset may be defined as a number of query requests 54 that theparticular master node 70 may process for the second client 12 atsubstantially the same time. For example, the particular master node 70may include a particular number of threads (e.g., thirty), each operableto receive and process a different query request 54 substantiallyconcurrently. A predetermined number of those threads may be assigned tothe second client 12 for receiving and processing query requests 54received from the second client 12. In this example, ten of thepredetermined number of query requests 54 are assigned to first client12 as the second subset. Although first subset and second subset aredescribed as being twenty and ten for purposes of this example, thefirst and second subsets may have any suitable size, according toparticular needs.

At step 604, a suitable component of system 14 or the particular masternode 70 may ration CPU cycles for the processing of query requests 54received by the particular master node 70 from first and second clients12, according to each of the first and second client 12 s' use of theirrespectively assigned subsets. For example, if the first client 12 iscurrently using all threads assigned to the first client 12 and thesecond client 12 is currently using none of the threads assigned to thesecond client 12, then all CPU cycles may be dedicated to processing thequery requests received by the first client 12, until such time as thesecond client 12 begins submitting query requests 54. In certainembodiments, it may be possible to ration CPU cycles at intermediatelevels of use by first and second clients 12. For example, if the firstclient 12 is currently using fifteen of the twenty threads assigned tothe first client 12 and the second client 12 is currently using five ofthe threads assigned to the second client 12, the CPU cycles that wouldordinarily be give to the currently idle threads (ten total idle threadsin this example) may instead be divided in any suitable manner betweenthe threads being used by the first and second clients 12.

At step 606, the first client 12 submits a particular query request 54to system 14. At step 608, a determination is made whether theparticular master node 70 is already processing a number of queryrequests 54 for the first client 12 that is greater than or equal to thefirst subset of query requests 54 assigned to the first client 12. Incertain embodiments, query receiving module 60 may make thisdetermination based on data received by communicating with theparticular master node 70. If it is determined at step 608 that theparticular master node 70 is currently processing less than the firstsubset of query requests 54 assigned to the first client 12, then atstep 610, the particular master node 70 may receive and process theparticular query request 54 submitted by the first client 12. Forexample, if one or more of the twenty threads of the particular masternode 70 that are assigned to the first client 12 are not currentlyprocessing query requests 54 for the first client 12 (and are nototherwise occupied), then at least one of the available threads mayreceive and process the query request 54 submitted by the first client12.

If it is determined at step 608 that the particular master node 70 iscurrently processing a number of query requests 54 that is greater thanor equal to the first subset of query requests 54 assigned to the firstclient 12, then at step 612, the particular master node 70 may notifythe first client 12 that there are no available threads. For example theparticular master node 70 may prompt the first client 12 to selectanother master node 70 or to resubmit the particular query request at alater time. Alternatively, the particular master node 70 or queryreceiving module 60 may automatically determine whether another masternode 70 has an available thread for processing the particular queryrequest 54 and may prompt the first client 12 to connect to thedetermined other master node 70 that has an available thread (or mayautomatically connect the first client 12 to the determined other masternode 70). Additionally or alternatively, query receiving module 60 mayautomatically attempt to resubmit the particular query request 54 to theparticular master node 70.

Although a particular method for managing the receipt and processing ofquery requests by one or more master nodes 70 of database system 14 hasbeen described with reference to FIG. 8, the present inventioncontemplates any suitable method for managing the receipt and processingof query requests by one or more master nodes 70 of database system 14in accordance with the present disclosure. Thus, certain of the stepsdescribed with reference to FIG. 8 may take place simultaneously and/orin different orders than as shown. As just one example, it may bedetermined whether one or more other master nodes 70 may receive andprocess the particular query request 54 received from the first client12 prior to determining whether the second client 12 is already tying upthe remaining availability of the particular master node 70. Moreover,system 10 may use methods with additional steps, fewer steps, and/ordifferent steps, so long as the methods remain appropriate.Additionally, certain steps may be repeated as needed or desired.

FIG. 9 illustrates an example method for processing requests to performone or more activities associated with a precompiled query 24 that arecommunicated by a particular master node 70 according to prioritiesassigned to the requests. In certain embodiments, the method describedwith reference to FIG. 9 may help system 14 to manage throughput inprocessing of query requests 54 received from one or more clients 12.Although particular numbers of master nodes and slave nodes 72 aredescribed with reference to FIG. 9, any or all of master nodes 70 andslave nodes 72 of system 14 may be capable of performing the methodwithout departing from the scope of the present invention.

At step 700, a priority is assigned to a request to perform one or moreactivities associated with a precompiled query 24. The priority mayinclude a high priority, a medium priority, a low priority, or any othersuitable priority, according to particular needs. In some examples, therequest may comprise a request package, as described above withreference to FIG. 2, and the priority may be identified by a priorityidentifier in the request package. In certain embodiments, the priorityidentifier may be a binary value for which zero represents a lowpriority and one represents a high priority, or vice versa.

The priority may be a user-specified priority in query request 54.Additionally or alternatively, the priority may be based on anidentification of the client that submitted the query request 54associated with the request package. For example, certain clients 12 maybe granted or assigned a higher priority than other clients 12, so thatquery requests 54 submitted by those clients 12 granted a higherpriority tend to be processed more quickly by system 14 than queryrequests 54 submitted by those clients 12 granted a lower priority.Additionally or alternatively, the priority may be based on the type ofquery request 54 and/or the type of activities associated with therequest package.

At step 702, a particular master node 70 may communicate the request ona communication channel, which may be determined as described above atleast with reference to FIG. 2. At step 704, a particular slave node 72receives the request communicated by the particular master node 70. Atstep 706, the particular slave node 72 may insert the request into anincoming queue 84 of the particular slave node 72, according to thepriority associated with the request. For example, the particular slavenode 72 may inspect the priority identifier in the request package, andif the priority identifier indicates that the request package has a highpriority, then the slave node 72 may insert the request package at thehead or substantially at the head of its incoming queue 84.

Additionally or alternatively, the particular slave node 72 may maintainseparate incoming queues 84, one incoming queue 84 being for highpriority requests and one incoming queue 84 being for low priorityrequests. In certain embodiments, the particular slave node 72 mayselect a request for processing from its high priority incoming queue 84before or more frequently than it selects a request for processing fromits low priority incoming queue 84. Although two incoming queues 84 aredescribed (one for high priority requests and one for low priorityrequests), the present invention contemplates the particular slave node72 maintaining a number of incoming queues 84, each corresponding to oneof a number of degrees of priority (e.g., high priority, mediumpriority, and low priority). Prioritizing requests received from masternodes 70 may enable may help system 14 to manage throughput inprocessing of query requests 54 received from one or more clients 12. Inparticular embodiments, prioritizing requests may help system 14 toincrease or maximize throughput for query requests 54 of higherimportance or priority.

Although a particular method for processing requests to perform one ormore activities associated with a precompiled query 24 that arecommunicated by a particular master node 70 according to prioritiesassigned to the requests has been described with reference to FIG. 9,the present invention contemplates any suitable method for processingrequests to perform one or more activities in accordance with thepresent disclosure. Thus, certain of the steps described with referenceto FIG. 9 may take place simultaneously and/or in different orders thanas shown. Moreover, system 10 may use methods with additional steps,fewer steps, and/or different steps, so long as the methods remainappropriate. Additionally, certain steps may be repeated as needed ordesired.

FIG. 10 illustrates an example method for returning results of a requestto perform one or more activities associated with a precompiled querycommunicated by a master node 70 from one or more slave nodes 72 to themaster node 72. Although a particular master node 70 and a particularslave node 72 are described as performing certain steps of the method,any or all of master nodes 70 and slave nodes 72 may be capable ofperforming substantially similar steps. It should also be understoodthat each master node 70 of system 14 may be processing multiple queryrequests on multiple threads of the master node 70, and may havecommunicated requests to perform one or more activities associated withone or more precompiled queries 54 to one or more slave nodes 72 ofsystem 14. Thus, each slave node 72 may have related or unrelatedresults for multiple master nodes 70 (or multiple related or unrelatedresults for a single master node 70), and each master node 70 may havemultiple requests for which the master node 70 is awaiting results fromone or more slave nodes 72.

At step 800, a particular slave node 72 receives from a particularmaster node 70 a request to perform one or more activities associatedwith a precompiled query 24. For example, the request may be a requestto perform an index read of one or more key parts 30 accessible to theparticular slave node 72. In certain embodiments, the request maycomprise a request package, as described above with reference to FIG. 2.The request may have been communicated by the particular master node 70over a communication channel, as a multicast message for example. Incertain embodiments, a substantially similar request may be received bymultiple slave nodes 72 on the same or a different communicationchannel.

At step 802, the particular slave node 72 processes the request byperforming at least a portion of the one or more activities of therequest to obtain one or more results for the request. For example, ifthe request is a request to perform an index read of one or more keyparts 30 accessible to the particular slave node 72, the slave node mayaccess the one or more key parts 30 to obtain results according to oneor more parameters included in the request.

At step 804, the particular slave node 72 may insert the results into anoutgoing queue 86 of the particular slave node 72. The outgoing queue 86may include results obtained by the particular slave node 72 for one ormore master nodes 70. For example, the particular slave node 72 may haveprocessed requests received from a number of master nodes 70, and theresults obtained for each of those requests may be stored in outgoingqueue 86. Alternatively, the particular slave node 72 may maintainseparate outgoing queues 86 for each master node 70 or store results inany other suitable manner.

At step 806, the particular slave node 72 may communicate arequest-to-send message to the particular master node 70. In certainembodiments, the request-to-send message may include an identificationof the particular master node 70, a transaction ID associated with therequest package to which the results obtained by the particular slavenode 72 correspond, time stamp information, or any other suitableinformation according to particular needs. The request-to-send messagemay be communicated to the particular master node 70 over a dedicatedflow-control channel. In embodiments in which the particular slave node72 has obtained results for multiple requests for multiple master nodes70, the particular slave node 72 may have sent multiple request-to-sendmessages to multiple master nodes 70, and the particular slave node 72may be waiting for a permission-to-send message from one of the masternodes 70 (including the particular master node 70) before sending anyresults.

At step 808, the particular master node 70 receives the request-to-sendmessage and stores it in queue 87. At step 810, the particular masternode 70 accesses queue and selects the request-to-send message sent bythe particular slave node 72. It should be appreciated that theparticular master node 70 may have processed other request-to-sendmessages in queue 87 between steps 808 and 810.

At step 812, the particular master node 70 determines whether theparticular slave node 72 is available to send the results. In certainembodiments, the particular master node 70 determines whether theparticular slave node 72 is currently busy sending results to anothermaster node 70 based on information communicated over the flow-controlchannel. The particular master node 70 (as well as all master nodes 70)may be listening on the flow-control channel for messages indicatingwhich slave nodes 72 are currently busy sending results to one or moremaster nodes 70.

If the particular master node 70 determines at step 812 that theparticular slave node 72 is available to send the results, then at step814, master node 70 may communicate a permission-to-send message to theparticular slave node 72, granting the particular slave node 72permission to send the results. As described above, thepermission-to-send message may be associated with a predefined timelimit, after which the permission is no longer valid. If that time limitexpires, the request-to-send message for which the permission-to-sendmessage was communicated may be reinserted in queue 87.

At step 816, in response to the permission-to-send message, theparticular slave node 72 may communicate the results to the particularmaster node 70, as described above with reference to FIG. 2. In certainembodiments, the results may comprise a result package, which mayinclude a transaction ID (e.g., at a high-level layer of the transportmechanism). Inclusion of the transaction ID may allow the particularmaster node 70 to match the results with the request that the mastercommunicated to the particular slave node 72, and/or with other resultpackages for the request.

If, at step 812, the particular master node 70 determines that theparticular slave node 72 is not available to send the results, then atstep 818, the particular master node 70 may access queue 87 for anotherrequest-to-send message to process. The particular master node 70 mayattempt to find a slave node 72, from which the particular master node70 has received a request-to-send message, that is not currently busysending results to another master node 70. Alternatively, if theparticular master node 72 has not received any other request-to-sendmessages, then the particular master node 70 may wait until it receivesanother request to send or determines that the particular slave node 72is available to send a permission-to-send message.

Although a particular method for returning results of a request toperform one or more activities associated with a precompiled querycommunicated by a master node 70 from one or more slave nodes 72 to themaster node 72 has been described with reference to FIG. 10, the presentinvention contemplates any suitable method for returning results inaccordance with the present disclosure. Thus, certain of the stepsdescribed with reference to FIG. 10 may take place simultaneously and/orin different orders than as shown. Moreover, system 10 may use methodswith additional steps, fewer steps, and/or different steps, so long asthe methods remain appropriate. Additionally, certain steps may berepeated as needed or desired.

Although the present invention has been described with severalembodiments, diverse changes, substitutions, variations, alterations,and modifications may be suggested to one skilled in the art, and it isintended that the invention encompass all such changes, substitutions,variations, alterations, and modifications as fall within the spirit andscope of the appended claims.

1. A database system for processing a query request, comprising: a firstmaster node running on at least one first processor for: communicating arequest to perform one or more activities associated with a precompiledquery over at least one communication channel of the database system,receiving a request-to-send message from a first slave node on a flowcontrol communication channel, wherein the flow control communicationchannel is accessible by a plurality of master nodes including the firstmaster node and another master node, listening, after receiving therequest-to-send message, to the flow control communication channel formessages between the first slave node and the another master node todetermine an availability of the first slave node at least based onwhether or not the first slave node is currently sending results to theanother master node, communicating a permission-to-send message to thefirst slave node, if it is determined from the listening on the flowcontrol communication channel that the first slave node is availablebecause it is not currently sending results to the another master node,wherein the permission-to-send message is valid for a predefined periodof time, and communicating the permission-to-send message to a secondslave node if it is determined from the listening on the flow controlcommunication channel that the first slave node is unavailable becauseit is currently sending results to the another master node; and aplurality of slave nodes running on at least one second processorcoupled to the first master node, at least a particular one of the slavenodes for: receiving over the communication channel the requestcommunicated by the first master node; performing at least a portion ofthe one or more activities associated with the request to obtain one ormore results for the request; communicating the request-to-send messageto the first master node indicating that the one or more results areavailable for communication to the first master node; and communicatingat least a portion of the one or more results to the first master nodein response to receiving the permission-to-send message from the firstmaster node.
 2. The system of claim 1, wherein the request comprises arequest package comprising a transaction ID for uniquely identifying therequest.
 3. The system of claim 2, wherein the request package comprisesan identification of the first master node that communicated the requestpackage.
 4. The system of claim 2, wherein each of the request-to-sendand permission-to-send messages comprise the transaction ID of therequest.
 5. The system of claim 1, wherein the first master nodecommunicates the permission-to-send message over the flow controlcommunication channel accessible to one or more other master nodes. 6.The system of claim 1, wherein the first master node reinserts therequest-to-send message in a queue if the results are not received bythe master node within the predefined period of time, the queue storingrequest-to-send messages received by the first master node.
 7. Thesystem of claim 1, wherein the first master node: receives at least aportion of the one or more results from the particular slave node; anddetermines, based at least on a transaction ID associated with the oneor more results, whether the one or more results are a duplicate of oneor more other results received by the first master node; and discardsthe results received from the particular slave node if the one or moreresults are determined to be a duplicate.
 8. The system of claim 1,wherein the at least one of the plurality of slave nodes determines asize of the portion of the one or more results based on a maximumcommunication size specified in the permission-to-send messagecommunicated by the first master node.
 9. A method for processing aquery request, comprising: communicating from a first master node arequest to perform one or more activities associated with a precompiledquery over at least one communication channel of the database system;receiving, at least at a particular one of a plurality of slave nodescoupled to the first master node, over the communication channel therequest communicated by the first master node; performing at theparticular slave node at least a portion of the one or more activitiesassociated with the request to obtain one or more results for therequest; communicating from the particular slave node a request-to-sendmessage to the first master node on a flow control communication channelindicating that the one or more results are available for communicationto the first master node, wherein the flow control communication channelis accessible by a plurality of master nodes including the first masternode and another master node; listening, after receiving therequest-to-send message, to the flow control communication channel formessages between the first slave node and the another master node todetermine, prior to communicating a permission-to-send message to theparticular slave node, an availability of the first slave node at leastbased on whether or not the particular slave node is currently sendingresults to the another master node; communicating the permission-to-sendmessage to the particular slave node if it is determined from thelistening on the flow control communication channel that the particularslave node is available because it is not currently sending results tothe another master node, wherein the permission-to-send message is validfor a predefined period of time; communicating the permission-to-sendmessage to a different slave node if it is determined from the listeningon the flow control communication channel that the particular slave nodeis not available because it is currently sending the results to theanother master node; and communicating from the particular slave node atleast a portion of the one or more results to the first master node inresponse to receiving the permission-to-send message from the firstmaster node.
 10. The method of claim 9, wherein the request comprises arequest package comprising a transaction ID for uniquely identifying therequest.
 11. The method of claim 10, wherein the request packagecomprises an identification of the first master node that communicatedthe request package.
 12. The method of claim 10, wherein each of therequest-to-send and permission-to-send messages comprise the transactionID of the request.
 13. The method of claim 9, comprising communicatingthe permission-to-send message over the flow control communicationchannel accessible to one or more other master nodes.
 14. The method ofclaim 9, further comprising reinserting the request-to-send message in aqueue if the results are not received by the first master node withinthe predefined period of time, the queue storing request-to-sendmessages received by the first master node.
 15. The method of claim 9,further comprising: receiving at the first master node at least aportion of the one or more results from the particular slave node;determining at the first master node, based at least on a transaction IDassociated with the one or more results, whether the one or more resultsare a duplicate of one or more other results received by the firstmaster node; and discarding at the first master node the one or moreresults received from the particular slave node if the one or moreresults are determined to be a duplicate.
 16. The method of claim 9,further comprising determining, by the particular slave node, a size ofthe portion of the one or more results based on a maximum communicationsize specified in the permission-to-send message communicated by thefirst master node.
 17. A computer-readable medium comprising programinstructions, which, when executed by a processor, cause the processorto perform a method for processing a query request, the methodcomprising: communicating from a first master node a request to performone or more activities associated with a precompiled query over at leastone communication channel of the database system; receiving, at least ata particular one of a plurality of slave nodes coupled to the firstmaster node, over the communication channel the request communicated bythe first master node; performing at the particular slave node at leasta portion of the one or more activities associated with the request toobtain one or more results for the request; communicating from theparticular slave node a request-to-send message to the first master nodeon a flow control communication channel indicating that the one or moreresults are available for communication to the first master node,wherein the flow control communication channel is accessible by aplurality of master nodes including the first master node and anothermaster node; listening, after receiving the request-to-send message, tothe flow control communication channel for messages between the firstslave node and the another master node to determine, prior tocommunicating a permission-to-send message to the particular slave node,an availability of the first slave node at least based on whether or notthe particular slave node is currently sending results to the anothermaster node; communicating the permission-to-send message to theparticular slave node if it is determined from the listening on the flowcontrol communication channel that the particular slave node isavailable because it is not currently sending results to the anothermaster node, wherein the permission-to-send message is valid for apredefined period of time; communicating the permission-to-send messageto a different slave node if it is determined from the listening on theflow control communication channel that the particular slave node is notavailable because it is currently sending the results to the anothermaster node; and communicating from the particular slave node at least aportion of the one or more results to the first master node in responseto receiving the permission-to-send message from the first master node.